bug with Natlink and Dragon16: natlink.playEvents and other functions

[77capr]

Hello,

I was using Natlink 5.5.4 with Dragonfly 0.0.35 for some time without issue. I recently upgraded to Dragonfly 1.0.0-rc2. I then started seeing this error (see also attachment) as soon as I spoke a command involving Dragonfly. I now upgraded to Natlink 5.5.6 without any change in behavior. I'm currently dead in the water with respect to using Natlink/Dragonfly.

Any ideas on how to get around this?

Thanks,

Al

MS Surface Pro 8 Windows 11 DNS 16 Python 3.10.8

[LexiconCode]

Sorry for your trouble.

1. Post the output of 'python -m pip list' from the nat link python interpreter.

2. Does downgrading dragonfly to 0.35.0 help?

[77capr]

Thanks for responding!

Yes, downgrading to dragonfly 0.35.0 (I mistakenly wrote 0.0.35 above) fixes the issue. Trying with dragonfly 1.0.0rc1 shows the same error as with 1.0.0rc2

Here's the pip output (sorry, I can't figure out how to format it monospaced):

python -m pip list

Package Version

---

appdirs 1.4.4 certifi 2023.5.7 charset-normalizer 3.2.0 comtypes 1.2.0 debugpy 1.6.7 decorator 5.1.1 dragonfly2 1.0.0rc2 dtactions 1.5.7 fissix 21.11.13 idna 3.4 json-rpc 1.15.0 lark-parser 0.8.9 MarkupSafe 2.1.3 modernize 0.8.0 natlink 5.5.6 natlinkcore 5.3.7 packaging 23.1 pip 23.3.1 pydebugstring 1.0.0.0 pyperclip 1.8.2 PySimpleGUI 4.60.5 pywin32 306 regex 2023.6.3 requests 2.31.0 setuptools 63.2.0 six 1.16.0 urllib3 2.0.3 Werkzeug 2.3.6

[LexiconCode]

Have you tried just a sample grammar/script from dragonfly?

Trying to rule out if there's something with strange with a your script that you might have loaded.

[77capr]

It looks like it's related to the following two lines in one of my scripts:

from dragonfly.actions.mouse import ButtonEvent
ButtonEvent.try_natlink = True

Using the Mouse class in Dragonfly 1.0.0 with these two lines seems to trigger the error, whereas in Dragonfly 0.35.0 it does not.

Not a big loss for me to not use these two lines. Feel free to close the issue if it's not worth digging into further.

Thanks for your quick assistance!

[drmfinlay]

This problem seems specific to DNS 16. I cannot reproduce it with DNS 15 and the same version of Natlink.

The new try_natlink Dragonfly feature, enabled by @77capr's code above, utilises Natlink's playEvents() function. That function looks to be causing the error.

Do you know if this also occurs with Dragonfly's keyboard actions, @77capr?

[77capr]

I tried with:

from dragonfly.actions.keyboard import Keyboard
Keyboard.try_natlink = True

and yes, then invoking dragonfly keyboard actions also trigger the same error

[drmfinlay]

Thank you, @77capr. I'm not sure if we'll be able to fix the problem. In the meantime, I'll update Dragonfly's documentation to mention this as a potential issue with the try_natlink feature.

[77capr]

👍 in any case, thanks for all the work on this software! It makes a difference in my life.

On Wed, Nov 8, 2023, 06:02 Dane Finlay @.***> wrote:

Thank you, @77capr https://github.com/77capr. I'm not sure if we'll be able to fix the problem. In the meantime, I'll update Dragonfly's documentation to mention this as a potential issue with the try_natlink feature.

— Reply to this email directly, view it on GitHub https://github.com/dictation-toolbox/natlink/issues/176#issuecomment-1801089874, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYBOO265CYYEOWANJU2BW3YDMG7XAVCNFSM6AAAAAA66SYLIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBRGA4DSOBXGQ . You are receiving this because you were mentioned.Message ID: @.***>

[quintijn]

I only read this now. It seems to boil down to playString (which can easily be avoided) and playEvents, which is probably more hard to tackle.

[drmfinlay]

This is clearly some sort of interface change in Dragon 16, since I can't reproduce the issue on Dragon 15. I don't think we can fix it.

I mentioned some time ago that there should be info in the documentation on what doesn't work in certain versions of Dragon. Perhaps we should revisit that idea? For this issue, a line in natlink.txt could be added:

playEvents( events ) This function is a more powerful version of playString which can play any sequence of events. ... Fails with an error on Dragon 16+.

[drmfinlay]

Certainly these functions can be avoided. It is a shame things like this are starting to break, however, since they have been stable for so long. I suppose we should be grateful our COM interface definitions from the late 90's still work at all.

I think this issue should remain open, since the problem affects both projects: natlink and natlinkcore. I'll change the title to something more sensible though.

[quintijn]

Avoiding these two functions natlink.playString and natlink.playEvents has been attempted via the __init__.py module of natlink.

1. natlink.playString is now performed with python functions
2. natlink.playEvents is disabled when Dragon 16 is running. Hopefully someone can write this function in python as well

We hope natlink will be stable, and does not cause ESP Errors (something to do with the Stack Pointer) any more.

We will try to release the natlink installer ASAP and mention it here.

[drmfinlay]

Okay then, sounds good. Perhaps I could take a look at writing the playEvents function in Python. I have a pretty good idea of how it works.

[drmfinlay]

The below function should do essentially what Natlink's playEvents() function does, minus the UI access privilege Dragon gives it. It doesn't work with some windows though. I'm not sure why.

import win32api
import win32gui

def playEvents(events):
    # Check that *events* is a list.
    if not isinstance(events, list):
        raise TypeError("events must be a list of 3-tuples")

    # Post each 3-tuple event to the foreground window.
    hwnd = win32gui.GetForegroundWindow()
    for event in events:
        if not (isinstance(event, tuple) and len(event) == 3 and
                all((isinstance(i, int) for i in event))):
            raise TypeError("events must be a list containing 3-tuples of"
                            " integers")
        message, wParam, lParam = event
        win32api.PostMessage(hwnd, message, wParam, lParam)

[quintijn]

Hi Dane, others,

I tried to implement your code in init.py of https://github.com/dougransom/natlink, branch "fordragon16", with a test routine at the bottom. It gives no mouse movement regrettably.

A complete installer on my computer (with Visual Studio Code) did not install correct. So several things to sort out!!

[drmfinlay]

Hi Quintijn,

Thanks for testing my code. Replacing this function is not as simple as I thought it would be. Probably it is better to use the Windows SendInput() function, as it is more reliable. It can send keyboard and mouse events.

I regret I can't debug the original runtime error myself. I notice some people online mentioning the /vmg MSVC compiler option. Maybe someone with Dragon 16 could try a build of natlink with that option?

[quintijn]

This issue is now closed, in favour of https://github.com/dictation-toolbox/natlink/issues/196