Open LexiconCode opened 4 years ago
thks
On Sat, Feb 1, 2020 at 4:39 PM LexiconCode notifications@github.com wrote:
Is your feature request related to a problem? Please describe. Credit goes to @davidlehub https://github.com/davidlehub for initial creation of idea and implementation thus far.
Dragonflies grammar exclusivity allows grammars to be exclusive over normal grammars. Exclusive grammars are only recognized.
This allows for a few things.
- Exclusive Grammars can override dragons built in grammars.
- Hopefully makes creating different modes like spelling mode and such easier to implement.
Describe the solution you'd like A system to manage grammar exclusivity. To what extent an exact implementation yet to be determined.
Preferably implemented in casters Hooks and Events system for minimal impact on core code.
Describe alternatives you've considered The only alternative is to disable all of the rules and even then dragons built in rules interfere.
Summarized quotes and resources from davidlehub from gitter.
davidlehub
I putted Exclusivity Manager in 1 file. (for people to test) 1.Download this "init.py" from https://drive.google.com/open?id=1BD8JojzPaROke_N9ATApBt74QC3bTaUv
- Put that file in Caster folder ("Documents\Caster")
davidlehub
Relate to "hooks", i suspect the "HookRunner" could be interesting. Maybe this could help: pictures https://drive.google.com/open?id=14Crr7BhV5vwxJHYm08NmNEPuUNLuPgzE
davidlehub
To make that system more useful. I had to: Make it works per app: Auto add "context app" rules to be exclusive, when foreground windows changed. Store/remember witch rules are been exclusives per app. Enable Dragon vocabularies when needed. etc.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dictation-toolbox/Caster/issues/738?email_source=notifications&email_token=ALNO3O3Y4AFSWWK22UN7RSLRAXTYFA5CNFSM4KOVTIXKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKLS2YQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALNO3O4IHM5YWGMUILC3SIDRAXTYFANCNFSM4KOVTIXA .
:) Also, maybe the videos? It could help peoples to have some idea how exclusiveness can be useful.
@davidlehub That would be a good idea. You can even update your current post with the videos.
Here they are: https://www.youtube.com/playlist?list=PLQ6G9K9v8sUR44kvrit9pFoSBvGPKgM-7
Again, sorry for my English end the slowness. My firs time self recording explaining some things :D
I found the following in \Dragonfly\dragonfly\grammar\grammar_base.py
Perhaps it could be used to enable/disable exclusive mode when switching between windows/applications?
def enter_context(self):
"""
Enter context callback.
This method is called when a phrase-start has been
detected. It is only called if this grammar's
context previously did not match but now does
match positively.
"""
def exit_context(self):
"""
Exit context callback.
This method is called when a phrase-start has been
detected. It is only called if this grammar's
context previously did match but now doesn't
match positively anymore.
"""
Also, If all Dragon commands are disabled when in exclusive mode, perhaps we should make a list of, and devise workarounds/pass-throughs for, any Dragon commands we wish to keep? Such as: "Go To Sleep" "Wake Up" (if also excluded) "Click [buttontext]" "Press [modifiers] [key]" If Mimic() won't work in exclusive mode?
I found the following in
\Dragonfly\dragonfly\grammar\grammar_base.py
Perhaps it could be used to enable/disable exclusive mode when switching between windows/applications?def enter_context(self): """ Enter context callback. This method is called when a phrase-start has been detected. It is only called if this grammar's context previously did not match but now does match positively. """ def exit_context(self): """ Exit context callback. This method is called when a phrase-start has been detected. It is only called if this grammar's context previously did match but now doesn't match positively anymore. """
Also, If all Dragon commands are disabled when in exclusive mode, perhaps we should make a list of, and devise workarounds/pass-throughs for, any Dragon commands we wish to keep? Such as: "Go To Sleep" "Wake Up" (if also excluded) "Click " "Press " If Mimic() won't work in exclusive mode?
Relate to that, there is also:
def _process_begin(self, executable, title, handle):
"""
Start of phrase callback.
*This usually is the method which should be overridden
to give derived grammar classes custom behavior.*
This method is called when the speech recognition
engine detects that the user has begun to speak a
phrase. This method is called by the
``Grammar.process_begin`` method only if this
grammar's context matches positively.
Arguments:
- *executable* -- the full path to the module whose
window is currently in the foreground.
- *title* -- window title of the foreground window.
- *handle* -- window handle to the foreground window.
"""
Example, this is what i did initially to detect >> foreground windows changed :
class grammExclusivenessCtrler_rule(MergeRule):
# mapping = {
# }
def _process_begin(self):
# ...
activeWinHnd = win32gui.GetForegroundWindow()
if activeWinHnd != gl.previousWinHnd:
""" Forground windows changed, so:
Make (or restore) a set of rules/grammars to be exclusive for the current forground window.
"""
# ...
gl.previousWinHnd = activeWinHnd
note the following:
(That’s why, since i don’t know how with python, i used an external tool, Autohotkey, to detect the "foreground window changed" EVENT, then tell Caster it happens. )
"Go To Sleep" "Wake Up" (if also excluded) "Click " "Press "
They are all excluded (all Dragon commands).
(Yeaa, our Dragon is not happy and turns a deaf ear, bcz we excluded him :-D )
As Solution, what i did:
In short:
In short:
Store the "state" of the current exclusiveness.
Disable exclusiveness of all grammar. It makes Dragon commands available.
Do the "mimic()" of a Dragon command.
Finally, switch back to the state stored at step 1.
I was thinking the same thing :) But was wondering how quickly exclusivity can be switched on and off?
Each time a rule's exclusiveness is changed, must it be unloaded and reloaded? Or disabled and re-enabled?
If listening for ShellMessages (HSHELL_WINDOWACTIVATED, HSHELL_RUDEAPPACTIVATED, HSHELL_WINDOWCREATED especially) is not possible with Caster, one possible way to switch context when the window changes would be to check the window handle on_end(), so that any time the window changed as a result of a voice command, exclusivity could be changed post voice command and be less impactful, and would only need to be changed on_begin() the times it was changed by keyboard or mouse or an application popped up/stole focus.
Or perhaps Caster could have its own accompanying Python script which runs on its own thread and is entirely separate, much like an AutoHotkey script, which would not be bound by threading issues and could listen for ShellMessages and notify caster to change context/exclusiveness and perform numerous other functions (pun intended :p) asynchronously, perhaps it would help with GUIs also :)
I was thinking the same thing :) But was wondering how quickly exclusivity can be switched on and off?
With the new caster system, it seems pretty fast. But event with the old one, when it has a noticeable delay (bcz of the waiting of the grammars to become exclusive), It doesn't that bad. But anyway, i think, it's preferable to wait a little bit (worst case 2 to 3 sec) and have Caster respond more accurately, then been frustrated by no needed vocabularies interfering and executing wrong commands.
Each time a rule's exclusiveness is changed, must it be unloaded and reloaded? Or disabled and re-enabled?
If i am not wrong, it disabled and re-enabled, except for the grammar/s of the CCR...
Pretty sure is the same with the new Caster system.
Or perhaps Caster could have its own accompanying Python script which runs on its own thread and is entirely separate, much like an AutoHotkey script, which would not be bound by threading issues and could listen for ShellMessages and notify caster to change context/exclusiveness and perform numerous other functions (pun intended :p) asynchronously, perhaps it would help with GUIs also :)
Oh yea, agreed :)
"Go To Sleep" "Wake Up" (if also excluded) "Click " "Press "
They are all excluded (all Dragon commands).
(Yeaa, our Dragon is not happy and turns a deaf ear, bcz we excluded him :-D )
As Solution, what i did:
- hmm, i think is more easy to explain with a screen view... i will post a video.
In short:
- Store the "state" of the current exclusiveness.
- Disable exclusiveness of all grammar. It makes Dragon commands available.
- Do the "mimic()" of a Dragon command.
- Finally, switch back to the state stored at step 1.
So, to use a Dragon command, for example "Go To Sleep", here is a "summary" of what i did (in the old Caster system): https://drive.google.com/open?id=160GLTsiEu3XDfVr-iCtDVgTecqAAvWkf
(you can open that drawing by clikcing on: So u can copy the texts in it, if needed.)
- That "_process_begin" function is called ONLY when the microphone start to dectecting some things (e.g., the user start to speak in to the microphone).
I think it should be fine though because that's how dragonfly monitors window changes for all grammars anyway and I haven't seen really any performance issues with it.
Also according to Dane Grammars shouldn't need to be unloaded before set_exclusiveness() calls take effect, although they do need to be loaded for that method to work.
I believe that means we just need to track which grammars we make exclusive.
Last Dane is thinking to include in rules the ability to define exclusiveness. That will work well for grammars of we always want to be exclusive when enabled.
@LexiconCode Thanks for passing on my messages :+1:
It would be pretty useful to have an exclusivity manager for enabling/disabling Dragon's built in rules like this!
It would be possible to implement a Rule.set_exclusiveness()
method for setting Dragonfly rules as exclusive/non-exclusive. It would be quite tricky to implement though and couldn't really work well with non-exported rules, which is how CCR usually works.
"Go To Sleep" "Wake Up" (if also excluded) "Click [button]" "Press [modifier] [key]"
"Go To Sleep" is also blocked by my free dictation interception "<text>":
, so I was able to devise and test this now, despite not yet having the ability to enable exclusiveness.
The following are working replacements:
import natlink
"Go To Sleep": Function(lambda: natlink.setMicState("sleeping") ),
"Mic Off": Function(lambda: natlink.setMicState("off") ),
As we cannot get the dictation while the microphone is asleep (inconvenient, but good for security and privacy), if "Wake Up": is also blocked, perhaps the only solution will be to disable all exclusiveness when going to sleep and re-enable it again upon waking (easily done with changeCallback())
It would be quite tricky to implement though and couldn't really work well with non-exported rules, which is how CCR usually works.
The way the current implementation works around this is disabling all rules then re-enabling exclusive only rules. There's not much added benefit to this outside of blocking Dragon.
I think the first step implementation here would be to enable exclusivity for all Caster grammars.
Going beyond that to work with CCR is going to be tricky. Honestly beyond my ability to pull off but I could imagine it something like this.
If the system worked as described there wouldn't be the penalty of reemerging everything in the first step when exiting exclusivity it would be just re-enabled
If the system worked as described there wouldn't be the penalty of reemerging everything in the first step when exiting exclusivity it would be just re-enabled.
:thumbsup: No reemerging = No delay, especially when the callback "_process_begin" is use to detect "foreground window" changes.
@davidlehub
For making exclusivity for all loaded Caster grammars. Simply placing grammar.set_exclusiveness(1)
right after grammar.load() as a proof of concept.
To be placed in the following lines MappingRules and MergeRule in GrammarManager.
Now that we know where to place the event
we can implement a hook
to do more advanced functionality.
I'm a bit short on time @lettersandnumbersgithub could fill you in on some of the other issues like Dragon sleeping
and so forth.
For making exclusivity for all loaded Caster grammars. Simply placing
grammar.set_exclusiveness(1)
right before grammar.load() as a proof of concept.For making exclusivity for all loaded Caster grammars. Simply placing
grammar.set_exclusiveness(1)
right after grammar.load() as a proof of concept.
:)
With this change alone, grammars remain exclusive and active even after putting Dragon to sleep (for which the workaround I described above is required) However I have written the following which appears to solve this problem:
"Go To Sleep": Function(lambda: GoToSleep() ),
def GoToSleep():
for grammar in get_engine().grammars:
grammar.set_exclusiveness(0)
natlink.setMicState("sleeping")
@davidlehub
@lettersandnumbersgithub
All right I have a working hook
an event
for exclusivity for all Caster loaded grammars. Note this does not include Dragonfly grammars as they are not loaded through Casters Grammar Manager.
If you try out the code above you'll need to start dns once. Then go to hooks.toml
and set ExclusiveHook = true
or say enable exclusive hook
then restart DNS again.
I'm curious to know the behavior when enabling the hook after starting caster compared to it's enabled on boot.
I'll take some feedback and questions here before making a pull request.
With this change alone, grammars remain exclusive and active even after putting Dragon to sleep (for which the workaround I described above is required) However I have written the following which appears to solve this problem:
"Go To Sleep": Function(lambda: GoToSleep() ),
def GoToSleep(): for grammar in get_engine().grammars: grammar.set_exclusiveness(0) natlink.setMicState("sleeping")
Just some think relate to it, on my mind now: in the function "GoToSleep", maybe storing the "state" of the current exclusiveness (for example, put in a list, all grammars, or rules, been exclusive). So when Dragon is "wake up", we restore the saved state.
Code would be some thing like:
def GoToSleep():
--> Store/save Things Currently Been Exclusive
for grammar in get_engine().grammars:
....
and then when, for example:
def WakeUp():
--> reStore Things that was been Exclusive
....
:)
@LexiconCode
If you try out the code above you'll need to start dns once. Then go to hooks.toml and set ExclusiveHook = true or say enable exclusive hook then restart DNS again.
how to try it? i click on "Create pull request"?
how to try it? i click on "Create pull request"?
Pull request are meant to request merging changes into a repository. In this context and the upstream Caster repository. Great for when you have a new feature you would like to include in Caster.
There's a few ways to go about doing this. Git - you'll need to add my repository has a remote. Once that's done you can check out the branches. See Git Extensions Docs
Zip - Download my branch through the zip file by going to my exclusive branch in my caster fork.
have error: AttributeError: 'NoneType' object has no attribute 'file'
Caster: castervoice is up-to-date
Error loading _caster from C:\Users\HP\Documents\Caster\_caster.py
Traceback (most recent call last):
File "C:\NatLink\NatLink\MacroSystem\core\natlinkmain.py", line 331, in loadFile
imp.load_module(modName,fndFile,fndName,fndDesc)
File "C:\Users\HP\Documents\Caster\_caster.py", line 65, in <module>
control.init_nexus(_content_loader)
File "C:\Users\HP\Documents\Caster\castervoice\lib\control.py", line 7, in init_nexus
_NEXUS = Nexus(content_loader)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\nexus.py", line 83, in __init__
self._load_and_register_all_content(rules_config, hooks_runner, transformers_runner)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\nexus.py", line 92, in _load_and_register_all_content
content = self._content_loader.load_everything(rules_config)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 57, in load_everything
rules = self._process_requests(rule_requests)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 106, in _process_requests
content_item = self.idem_import_module(request.module_name, request.content_type)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 92, in idem_import_module
return fn()
File "C:\Users\HP\Documents\Caster\castervoice\rules\core\utility_rules\window_mgmt_rule.py", line 20, in get_rule
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\rule_details.py", line 31, in __init__
self._filepath = RuleDetails._calculate_filepath_from_frame(stack, 1)
File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\rule_details.py", line 37, in _calculate_filepath_from_frame
filepath = module.__file__.replace("\\", "/")
AttributeError: 'NoneType' object has no attribute '__file__'
-- skip unchanged wrong grammar file: C:\Users\HP\Documents\Caster\_caster.p
(I used Git to have files localy.)
have error: AttributeError: 'NoneType' object has no attribute 'file'
delete window_mgmt_rule.pyc
I'm curious to know the behavior when enabling the hook after starting caster compared to it's enabled on boot.
Test result:
[+] Mannulaly make "ExclusiveHook = true", in the hooks.toml, exclusiveness works as expected.
[-] When speech "disable exclusive hook", the spec is recognised, but nothing change: exclusiveness is still actived/enabled. In hooks.toml, "ExclusiveHook " still = true. Even after restart Dragon.
[-] After manually change "ExclusiveHook = false", in the hooks.toml , then run Dragon, then speech "enable exclusive hook", the spec is recognised, but nothing change: exclusiveness is still deactived/disabled. "ExclusiveHook " still = false. Even after restart Dragon.
Okay that's pretty much what I expected. There's definitely more work to be done but it's a start. Some enhancements need to be made to the event and hook system for this to become robust.
Hi guys :) I wonder if there is already a function (or place in the code) that detect foreground/active window changed?
what i mean is a LOGIC that detect it. Not the methodes, like get_active_window_title , in castervoice\lib\utilities.py
@davidlehub you mean what makes the method through dragonfly?
I don't think there's a way through the engine to expose when there's a context change.
how can i use/refer this function def _change_rule_enabled(self, class_name, enabled, tail=True)
(witch is in '\castervoice\lib\ctrl\mgr\grammar_manager.py'), other way then _NEXUS._grammar_manager._change_rule_enabled(iRclassName, False)
?
@davidlehub you mean what makes the method through dragonfly?
I don't think there's a way through the engine to expose when there's a context change.
OK, then i am gonna try to create one. In fact gonna use a method i used on old old caster :)
how can i use/refer this function
def _change_rule_enabled(self, class_name, enabled, tail=True)
(witch is in '\castervoice\lib\ctrl\mgr\grammar_manager.py'), other way then_NEXUS._grammar_manager._change_rule_enabled(iRclassName, False)
?
Yes you can use this to the event system. The printer hook makes use of this.
Ahh, supper!
Can we have 2 hooks with a same returned value of def get_pronunciation(self):
?
for example, having 2 hooks with same pronuncation "A": ` class hook1(BaseHook):
...
def get_pronunciation(self):
return "A"
class hook2(BaseHook):
...
def get_pronunciation(self):
return "A"
`
Can we have 2 hooks with a same returned value of
def get_pronunciation(self):
?for example, having 2 hooks with same pronuncation "A": ` class hook1(BaseHook):
... def get_pronunciation(self): return "A"
class hook2(BaseHook):
... def get_pronunciation(self): return "A"
`
No that is not possible as think of it like a grammar in the sense of need its unique Pronunciation.
Hehe, little weird thing just happen with an GUI wx:
A GUI button is auto click when i switch windows/apps!
The funny is, this weird thing could be useful for detecting a foreground window has changed. But for now, the button pressed event should be fired only with clicked with a mouse.
To see that, check repository: branch 'NewExclusiveness', (the gui is at the end of "_caster.py")
Also try, when the second button ("click to On/Off") is pressed, then switch to an other window, the first button is auto focused and pressed. Event if the Gui is minimized.
That's a creative workaround! However I think the best way to proceed is to create a function that monitors the foreground window in dragonfly. Initially this would be first for Windows and then other implementations specific to each OS.
This article has some low-level windows information that can be used to create a function to monitor processes. it lays out four steps to create this method. Fortunately we only need to be concerned with the first as the rest are already implemented in dragonfly.
I think i won't use that GUI, for now, bcz i don't have the control on it (button should only press when i click on it, not auto click...).
but thanks :) >That's a creative workaround!
to create a function that monitors the foreground window in dragonfly.
hmm, since i don't know how to to it in "dragonfly", for now i think i'm gonna try to do it(the detection of foreground changed) where it detect "Speech start detected." (logged message):
Hey @LexiconCode, again: things are modifying in my code , like was happen in utility.py. Probably by accidentally, but... that never happen strangely like that before. I keep an eye on it, if happen again, i will report it again.
Funny... :)
@davidlehub keep working on what you're doing and I'll see what I can come up with.
About Git PUSH, what about the files in my "AppData\Local\caster": are they upload to github too? I guess is Yes.
About Git PUSH, what about the files in my "AppData\Local\caster": are they upload to github too? I guess is Yes.
nope
Hi guys, is the order of elements in the list '_enabled_ordered' (in side rules.toml) important? Or it can be in any order, like elements of a set()?
The enabled ordered is actually important. Editing that field directly is not recommended by hand or programmatically (outside of letting Caster manage it).
So in case you guys wanna try/test the current progress of the exclusiveness thing:
Edit this file 'C:\Python27\Lib\site-packages\dragonfly__main__.py':
--- inside the def _on_begin():
function, add:
`
from castervoice.exclusiveness.Notify_on_begin_fromDragonFly import Notify_on_begin_fromDragonFly
Notify_on_begin_fromDragonFly()
` --- So it looks like this:
` def _on_begin(): print("Speech start detected.")
from castervoice.exclusiveness.Notify_on_begin_fromDragonFly import Notify_on_begin_fromDragonFly
Notify_on_begin_fromDragonFly()
`
Note: My first main goal is to make things works. Very probably, the codes are not well structure, optimized, not implementing the new hooks-event system, etc. I adapt my codes from old caster version. :)
The enabled ordered is actually important. Editing that field directly is not recommended by hand or programmatically (outside of letting Caster manage it).
ho oh... hmm, will take care of this later later. bcz for now my exclusiveness system affect that variable (_enabled_ordered). I think is this function '_NEXUS._grammar_manager._change_rule_enabled(..)' that modify that variable.
That should be okay as I believe that's what internally Caster uses to manage that variable.
Is your feature request related to a problem? Please describe. Credit goes to @davidlehub for initial creation of idea and implementation thus far.
Dragonfly grammar exclusivity allows grammars to be exclusive over normal grammars. Exclusive grammars are only recognized.
This allows for a few things.
Describe the solution you'd like A system to manage grammar exclusivity. To what extent an exact implementation yet to be determined.
Preferably implemented in casters
Hooks and Events
system for minimal impact on core code.Describe alternatives you've considered The only alternative is to disable all of the rules and even then dragons built in rules interfere.
Summarized quotes and resources from davidlehub from gitter.
davidlehub
davidlehub
davidlehub