Modifier key timing over RDP/VNC connections

[Versatilus]

When using Dragonfly over a VNC or remote desktop connection it appears to have timing issues with modifier keys. For example, when using Notepad, using a Dragonfly command to activate the file menu ("a-f") will instead simply print the letter "f". Similarly, using Dragonfly to format text for variable names such as "ThisIsAnExample" results in "tHisiSaNeXample".

My personal workaround for this is to edit the timeouts set in keyboard.py to simulate a typing speed of approximately 150 words per minute.

I've encountered this problem both with UltraVNC and TightVNC over a wireless LAN.

[kendonB]

I'm also having this problem using Dragon NaturallySpeaking 13 On Windows 10 with a remote connection to Windows server 2008. @Versatilus can you share exactly what you did?

[kendonB]

I have tried setting the on_events, off_events, and events timeouts to both very high or very low numbers and nothing seems to help. I am specifically having trouble with capitalizing and symbols on the number keys "ThisIsAnExample" comes out as "thisisanexample". "prekris" comes out as "90".

However, "queue lease 5", "queue ross 5", etc are very consistent.

[nihlaeth]

I am not sure this is the same issue, but I've experienced something similar using aenea to connect to a Linux computer. I have a password grammar that takes a long time because of decryption, and I have to make sure that the encrypted part of the password is typed before I dictate anything else or everything will get garbled.

Can this have anything to do with natlink being multi threaded? I would expect there to be a queue, so commands would execute sequentially, but I haven't found its location yet.

[chilimangoes]

That's an interesting thought. In both cases, the issue seems to be with events/actions that are processed very fast on one side of the connection and then subsequent events/actions being sent before any confirmation is received that the prior action was completed. In the case of modifier keys over RDP etc., that message handling happens at the OS and RDP level, so there's not much we can do about it other than adjust the typing delays (I'm pretty sure there's no way for Dragon to know when a keyboard message has been handled by the target app). In the case of Aenea though, it should be possible to send a confirmation to the server when an action has been processed by the client. But I don't use Aenea, so I'm not sure where you'd need to look for a queue, or to implement one if it doesn't exist.

Regarding the current issue, there was some relevant discussion back in 2015 that started with this comment on Caster's Gitter channel and then picked up again here. @Versatilus also posted a workaround in that thread, but I'd like to see a change to Dragonfly that allows the typing delays to be modified at run time so that restarting Dragon isn't needed.

[nihlaeth]

I'd like to see a change to Dragonfly that allows the typing delays to be modified at run time so that restarting Dragon isn't needed.

That should be possible to implement in your own grammars, after all, it's only strings that define this. As for a global option that will work for all grammars, is this repository still being maintained? I haven't seen any merged pull requests in a while.

[chilimangoes]

That should be possible to implement in your own grammars, after all, it's only strings that define this.

It is, yes. The problem is that typing has to be slowed down considerably to work over a remote desktop connection. And modifying a ton of commands, spread across dozens of grammars, to support this would be a PITA and kind of a hackish workaround IMO. Implementing it in Dragonfly would allow us to globally set a slower typing speed when connected to a remote computer, and then change back to the default when used locally.

As far as whether this repo, it seems like it's being at least somewhat maintained. @t4ngo has made some commits in the last year and commented on an issue as recently as a few days ago.

[kendonB]

Reporting on my progress: I tried modifying my dragonfly installation using this: https://gist.github.com/Versatilus/bec104228952b97e54fe

I now find that commands that use the Text function seem to work fine, while commands to use the Key function still seem to fail when being sent over Remote Desktop connection.

[kendonB]

Actually, it seems that the Key function only fails for dragonfly's built in keys that require modifiers. For example, Key("s-8") works fine for "*" but Key("asterisk") fails.

[chilimangoes]

Try adding some conditional print statements in keyboard.py -> get_keycode_and_modifiers to check what timeouts are being used for the Typable(s) generated for the asterisk.

[kendonB]

I need help :/

I added these to punctuation.py in caster:

"alternative":                        R(Key("s-2"), rdescript="At sign"),
"atty":                                  R(Key("at"), rdescript="At sign"),

And the print statement at the bottom to get the modifiers but it doesn't work:

def get_keycode_and_modifiers(cls, char):
        if isinstance(char, str):
            code = windll.user32.VkKeyScanA(c_char(char))
        else:
            code = windll.user32.VkKeyScanW(c_wchar(char))
        if code == -1:
            raise ValueError("Unknown char: %r" % char)

        # Construct a list of the virtual key code and modifiers.
        modifiers = []
        if   code & 0x0100: modifiers.append(cls.shift_code)
        elif code & 0x0200: modifiers.append(cls.ctrl_code)
        elif code & 0x0400: modifiers.append(cls.alt_code)
        code &= 0x00ff

        if char == "@": print modifiers 

        return code, modifiers

[chilimangoes]

When you say it doesn't work, I assume you mean nothing happens?

First rule of troubleshooting: check your assumptions. Try printing all chars passed in to the function to a) make sure print calls from here are being output to the NatLink window like we expect and b) verify the type and value of char. You might also try printing out all modifiers too to make sure that variable is being printed the way you expect.

[kendonB]

OK, I think I did a better job (I am both a Python and dragonfly noob):

def get_keycode_and_modifiers(cls, char):
        if isinstance(char, str):
            code = windll.user32.VkKeyScanA(c_char(char))
        else:
            code = windll.user32.VkKeyScanW(c_wchar(char))
        if code == -1:
            raise ValueError("Unknown char: %r" % char)

        # Construct a list of the virtual key code and modifiers.
        modifiers = []
        if   code & 0x0100: modifiers.append(cls.shift_code)
        elif code & 0x0200: modifiers.append(cls.ctrl_code)
        elif code & 0x0400: modifiers.append(cls.alt_code)
        code &= 0x00ff

        if char == "@": 
            print "type of modifiers is " + str(type(modifiers))
            print "value of modifiers is "
            print '\n'.join(str(p) for p in modifiers)             
            print "type of char is " + str(type(char))
            print "value of char is " + str(char)
            print "value of code is " + str(code)
            print "type of code is " + str(type(code))

        return code, modifiers

Then, the following gets printed on loading:

type of modifiers is <type 'list'>
value of modifiers is 
16
type of char is <type 'str'>
value of char is @
value of code is 50
type of code is <type 'int'>
type of modifiers is <type 'list'>
value of modifiers is 
16
type of char is <type 'str'>
value of char is @
value of code is 50
type of code is <type 'int'>
type of modifiers is <type 'list'>
value of modifiers is 
16
type of char is <type 'str'>
value of char is @
value of code is 50
type of code is <type 'int'>

When I try and repeat the commands, nothing happens, but I guess these only get called at load time. All the inputs look identical, so the difference doesn't seem like it's here?

[kendonB]

I also tried printing all chars that go through here. Here is the output from that around the bit where the punctuation characters are generated:

6
6
7
7
8
8
9
9
!
!
@
type of modifiers is <type 'list'>
value of modifiers is 
16
type of char is <type 'str'>
value of char is @
value of code is 50
type of code is <type 'int'>
#
$
%
^
&
&
*
*

[chilimangoes]

So from what I remember when I looked into this back in 2015, you're right about the Typables for Keys getting created once at load time. The next step would be to do some similar debugging in the Typable class methods to see what values are being passed in for the timeout parameters. You would then want to find where these Typable instances are being created from - IOW where is Keyboard.get_typable being called from. Running a grep on the dragonfly code base would be a good way to find where it's being called from. If you're on Windows, AstroGrep is a good, easy to use grep tool for this. Alternately, you might also want to do some debugging of the send_keyboard_events function to see what timeouts are being passed in for key codes and modifiers for things like @ versus the timeouts passed in for capital letters used in a Text action.

[kendonB]

I'm going to go with a workaround for now of just changing everything to use the Text function.