Some commands on the new keyboard emulator do not work

[jdcai]

Describe the bug When I use commands with control + shift like control + shift + f it does not work. It is registering that the command has been said but it does not do control + shift + f. It seems to do something else. When I say control + shift + b in chrome it seems to do something that causes my screen to flash instead of what the command is supposed to be. If I say queue + b though it will toggle the bookmark bar properly in chrome.

Another issue is when I say control + slash it does not work and gives me this error:

press modifiers plus buttons from button_dictionary_1, non-repeatable
ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''

Another issue is that release does not work for me. I can say hold control but if I say release control the command is not recognized.

System:

• OS: Windows 10
• Program: chrome
• Version: 1.6.16 Caster master branch

[LexiconCode]

Thank you for opening up an issue. Let's see what we can do.

Have you enabled the Google Chrome Rule by saying "enable Google Chrome" an "enable Keyboard"?

Notes The keyboard grammar for keyboard emulation. https://caster.readthedocs.io/en/latest/readthedocs/Caster_Commands/Keyboard/

Definitions https://caster.readthedocs.io/en/latest/readthedocs/Rule_Construction/Taxonomy_of_a_Rule/ Spec is the voice command to trigger an action Action is the desired result could be keypress, text, a function and so forth Command represents the spec and action

Desired Commands

• Action control + shift + f, Spec control shift foxy Does nothing in chrome even when manually pressed. Works in other applications such as VS code

• Action control + shift + b, Spec control shift brov Successfully toggles chrome favorites bar

• Action control + slash Spec control slash I can confirm your issue with control + slash produces Execution failed: Keyboard interface cannot type this character: ''

@jdcai Let me know if what I've posted above is helpful. if there's any other commands include the keyboard shortcut you expect and the phrase/spec you're using to trigger it.

[LexiconCode]

Another issue is that release does not work for me. I can say hold control but if I say release control the command is not recognized.

With my testing below holding/releasing keys works correctly for single modifiers Such as control but not multiple such as control shift.

So I test this as well in VS code. If control + F is pressed the find modal box appears.

1. hold control # nothing visually happens as expected
2. press foxy # dialog box appears effectively emulating control + F
3. release control # nothing visually happens as expected
4. press foxy # emulates pressing the f but does not control + F. Therefore control has been released successfully

Now the same procedure control + shift + f which requires an investigation.

1. Highlight some text 1.1 hold control shift # nothing visually happens as expected
2. press foxy # sidebar find dialog box does not appear as expected.
3. release control shift
4. shift control foxy

@jdcai can you try the steps above?

@kendonB The output for the keyboard emulation doesn't inform what keys are pressed. press modifiers plus buttons from button_dictionary_1, non-repeatable This could be improved and passing the keys into rdescript

[jdcai]

I have both rules enabled.

Action control + shift + f, Spec control shift foxy Doesn't seem to work in VSCode for find in files. It does work in notepad++ though for find in files.

Action control + shift + n, Spec control shift novakeen Doesn't seem to work in Chrome for opening incognito window.

Action control + shift + P, Spec control shift prime Works in Firefox for opening private browsing

I realized that I had this transformer which is causing release to become releft. How would I change it so that it doesn't affect release?

<<<SPEC>>>
sauce -> up
dunce -> down
lease -> left
ross -> right
<<<NOT_SPECS>>>
sauce -> up
dunce -> down
lease -> left
ross -> right

Repeating your steps I have the same results by saying releft instead of release. When I try to do hold on multiple like hold control shift I see this error: ERROR:action.exec:Execution failed: Cannot use direction with modifiers..

[LexiconCode]

Action control + shift + f, Spec control shift foxy Doesn't seem to work in VSCode for find in files. It does work in notepad++ though for find in files.

VSCode make sure text is highlighted

Action control + shift + n, Spec control shift novakeen Doesn't seem to work in Chrome for opening incognito window.

That is is the correct shortcut However even on my Windows 10 machine control + shift + n does not seem to open up a new private tab even when pressing keys. is it the same for you? If it doesn't work with the keyboard it definitely will work to dragonfly.

I realized that I had this transformer which is causing release to become releft. How would I change it so that it doesn't affect release?

Oh that's an interesting edge case as transformers do replace substrings. That has its pros and cons obviously.

    <<<SPEC>>> (for specs only)
    <<<EXTRA>>>
    <<<DEFAULT>>>
    <<<NOT_SPECS>>> (for extras and defaults, but not specs)
    <<<ANY>>> (for specs, extras, and defaults).

If you are to keep your transformers <<<NOT_SPECS>>> that would keep releft from happening. I would have to review the code to remember how many specs include the directional keys.

[LexiconCode]

Action control + shift + n, Spec control shift novakeen Doesn't seem to work in Chrome for opening incognito window.

Specifically why control shift N and does not work is because it's already assigned within DNS/DPI. Removing thatshortcut will fix the issue.

[jdcai]

I tried it on another computer using Kaldi with debugging messages and looks like it's doing control+windows+shift instead of control+shift. I have the same things happening on this computer and on my other computer which uses dragon. https://superuser.com/questions/1127463/what-does-ctrlwinshiftb-do-in-windows Looks like control + windows + shift + b restarts the graphic driver which was why I was seeing my screen flash black when saying control + shift + b.

DEBUG:kaldi.compiler:parse_output(u'#nonterm:rule25 control shift vest #nonterm:end')
Level 12:engine:Alignment (word,time,length): [(u'#nonterm:rule25', 0, 0), (u'control', 2880, 12480), (u'shift', 15360, 10560), (u'vest', 25920, 13440), (u'#nonterm:end', 39360, 0)]
Recognized: control shift vest
DEBUG:action.exec:Executing action: RegisteredAction() ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
press modifiers plus buttons from button_dictionary_1, non-repeatable
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-v'
DEBUG:engine:rule execution time 3 ms
DEBUG:engine:dragonfly parse time 0 ms
Level 15:engine:End of phrase: eer=0.00 conf=nan, rule KaldiRule(25, g27::keyboard), u'control shift vest'

[LexiconCode]

Just so you're aware we have other tools as well.

Run_Caster_Test_Engine.bat

It can be run without a speech recognition engine. This is another way that helps differentiate between issue with the code or a misrecognition from the speech recognition backend.

[kendonB]

I can't reproduce this using the test engine:

*- Starting Caster v 1.6.16 -*
INFO:command:Calls to mimic() will be delayed by 3.00 seconds as specified
INFO:command:Enter commands to mimic followed by new lines.
control shift char
ERROR:command:Mimic failure for words: control shift char
enable keyboard
INFO:command:Mimic success for words: enable keyboard
control shift char
cs-c
INFO:command:Mimic success for words: control shift char
control shift windows char
cws-c
INFO:command:Mimic success for words: control shift windows char

I changed the rdescript as @LexiconCode suggested

[kendonB]

Also can't reproduce using Kaldi on Windows 10 x64 with Python38.

@jdcai can you try with a fresh user folder by editing Run_Caster_Kaldi to something like this?

@echo off
echo Running Kaldi from Dragonfly CLI

set currentpath=%~dp0
echo %currentpath%
echo Add psutil to requirements.txt
TITLE Caster: Status Window
set CASTER_USER_DIR=C:\test_caster_user
C:\Python38\python.exe -m dragonfly load _*.py --engine kaldi  --no-recobs-messages

pause 1

[LexiconCode]

I can't reproduce this using the test engine:

I can't reproduce this either. Looking at the debugging output that he provided despite the phrase control shift vest It's action is indeed being recognized as 'cws-'.

DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-v'

@jdcai

Test live dragonfly grammars

1. extract testme.zip to Documents\Caster (testme.py should be in the same directory as _caster.py)
2. open command prompt Documents\Caster
3. python -m dragonfly load testme.py --engine kaldi (this will load only testme.py and not the rest of caster). testme.py is built with only dragonfly for testing purposes.
4. Test out the following Specs and see if they register correctly in the debug output.
   • control shift brov
   • control windows shift brov
   • test control windows shift brov
   • test control shift brov
5. close out the command prompt Window

Test text engine dragonfly grammars

Text Engine Instructions The text engine does not rely in any speech recognition engine and helps us differentiate issues with code versus issues with the speech recognition backend.

Type commands to emulate as if they are being dictated by voice. lowercase mimics commands, UPPERCASE mimics free dictation Upper and lowercase words can be mixed e.g say THIS IS A TEST Edit the --delay 3 to change command delay in seconds. The delay allows user to switch to the relevant application to test commands

1. Reopen command prompt and run the text engine python -m dragonfly test testme.py -l DEBUG --delay 3
2. type the following one phrase at a time:
   • control shift brov
   • control windows shift brov
   • test control windows shift brov
   • test control shift brov

Test text engine caster grammars

1. Backup and delete your castor user directory.
2. Reopen command prompt and run the text engine python -m dragonfly test _caster.py -l DEBUG --delay 3
3. type the following one phrase at a time:
   • control shift brov
   • control windows shift brov

Depending on the results above would you be willing to connect over gitter and perhaps we can arrange a time for me to see remotely what this looks like first-hand?

@kendonB However Spec control slash is a different bug that is reproducible. Execution failed: Keyboard interface cannot type this character: ''

[jdcai]

I tried using a different user folder as @kendonB suggested using your Run_Caster_Kaldi.bat code and I still have the cws problem.

@LexiconCode I ran your commands using kendonB's code by changing the python line to your lines. I'm using Python 2.7

@echo off
echo Running Kaldi from Dragonfly CLI

set currentpath=%~dp0
echo %currentpath%
echo Add psutil to requirements.txt
TITLE Caster: Status Window
set CASTER_USER_DIR=C:\test_caster_user
python -m dragonfly test _caster.py -l DEBUG --delay 3

pause 1

Test Live Dragonfly Grammars The specs work as expected.

Test text engine dragonfly grammars The specs work as expected.

Test text engine caster grammars control shift brov still does cws-b instead of cs-b. Debug message below:

DEBUG:grammar.decode:               rollback: Choice(..., name='button_dictionary_1')
DEBUG:grammar.decode:                  attempt: Compound(u'brov')
DEBUG:grammar.decode:                     attempt: Literal([u'brov'])
DEBUG:grammar.decode:                     success: Literal([u'brov'])
DEBUG:grammar.decode:                      -- Decoding State: 'control shift brov >> '
DEBUG:grammar.decode:                  success: Compound(u'brov')
DEBUG:grammar.decode:               success: Choice(..., name='button_dictionary_1')
DEBUG:grammar.decode:            success: Sequence(...)
DEBUG:grammar.decode:         success: Compound(u'<modifier> <button_dictionary_1>')
DEBUG:grammar.decode:      success: Alternative(...)
DEBUG:grammar.decode:   success: Keyboard(keyboard)
DEBUG:action.exec:Executing action: RegisteredAction() ({'button_dictionary_1': 'b', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'brov'], 'modifier': 'cws-'})
press modifiers plus buttons from button_dictionary_1, non-repeatable
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'b', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'brov'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-b'
INFO:command:Mimic success for words: control shift brov

[kendonB]

@jdcai @LexiconCode I can reproduce with "control shift brov"

[kendonB]

So it's something to do with which spec. This is on the text engine

[LexiconCode]

So it's something to do with which spec. This is on the text engine

Hmm I agree. I believe both of us as far tested under Python 3 time to revisit with 2.

I do not think c-shift is valid. spec control shift

DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'shift', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift'], 'modifier': 'c-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'c-shift'
INFO:command:Mimic success for words: control shift

[LexiconCode]

So it's something to do with which spec. This is on the text engine

My time will be limited over the next 3 1/2 days as I have overnight shifts. At least we have a start.

[LexiconCode]

@kendonB when does the following work for you and does it emulate the same behavior as your original code?

I haven't figured out why this works yet in my limited testing due to time constraints.

def cat_spec_and_reverse(s1, s2): 

return ("(" + s1 + " " + s2 + ")" + " | " + "(" + s2 + " " + s1 + ")") 

 

def cat_spec_and_reverse_3(s1, s2, s3): 

action_cat_spec_and_reverse = cat_spec_and_reverse(s1, s2) 

return (action_cat_spec_and_reverse + s3 + " | (" + s1 + " " + s3 + " " + s2 + ")" + " | (" + s2 + " " + s3 + " " + s1 + ")" + " | (" + s3 + " " + s2 + " " + s1 + ")")

[kendonB]

It doesn't emulate the behaviour of the original code - but it may correct a bug in the original code! It also works for me (for a few examples at least) making that change - I will still have to check it properly when I'm at my linux computer (python debugging is easier there). I'll be offline for 3 days then will look. Feel free to push this fix if you find it addresses the bugs.

[kendonB]

I saw the control shift char bug both with and without the function change. It may be intermittent...

[jdcai]

The pull request fixes the issue with cws. There are also two other issues found on the keyboard commands.

control + slash gives this error:

press modifiers plus buttons from button_dictionary_1, non-repeatable
ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''

hold control shift gives this error: ERROR:action.exec:Execution failed: Cannot use direction with modifiers..

[LexiconCode]

The pull request fixes the issue with cws. There are also two other issues found on the keyboard commands.

control + slash gives this error:

press modifiers plus buttons from button_dictionary_1, non-repeatable
ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''

hold control shift gives this error: ERROR:action.exec:Execution failed: Cannot use direction with modifiers..

is that with this PR or without it?

[jdcai]

is that with this PR or without it?

Happens with and without the PR

[LexiconCode]

is that with this PR or without it?

Happens with and without the PR

@jdcai This should be fixed in the latest master.

[jdcai]

Oh I only had the pull request code. I updated to master and control + slash works now. hold control shift command was removed?

[LexiconCode]

Awesome.

hold control shift command was removed?

That should not have been removed.

[LexiconCode]

hold control shift command was removed?

Sigh no it's not working as intended. https://github.com/dictation-toolbox/Caster/issues/877

[kendonB]

I didn't think that I had ever implemented "hold control shift" @jdcai did that command really work as you expected before? The current code I believe only works for single keys

[kendonB]

What's the use case for "hold control shift"?

[kendonB]

it should work now with "hold control hold shift" btw

[kendonB]

I'd prefer not to allow for multiple keys with hold and release. That's just more scope for misrecognition with very limited use case

[LexiconCode]

I'd prefer not to allow for multiple keys with hold and release. That's just more scope for misrecognition with very limited use case

That makes sense, one thing I would like to do in a future PR is register the keys that are held down. that way they can be an generic release held keys without specifying the individual keypresses.

[LexiconCode]

What's the use case for "hold control shift"?

I could see use case for this when wanting to perform multiple command keypresses held. For instance if you hold "hold control shift" followed by directional keys you can say select text most programs. Obviously dedicated commands are better for this but for developing or troubleshooting these would be nice. Most importantly it provides consistency with dictating commands.

If I can find a convenient way and to add - Symbols when they're used as a modifier key this should be pretty simple to implement. Does that sound okay?

[kendonB]

Most importantly it provides consistency with dictating commands.

Not sure what you're referring to here?

If I can find a convenient way and to add - Symbols when they're used as a modifier key this should be pretty simple to implement.

Also don't understand this sorry