Proposal: Change Natlink to work exclusively out of process

[drmfinlay]

As discussed over e-mail and in the developer chat, changing Natlink to work exclusively in normal Python processes would resolve a number of open issues and simplify the software (natlink and natlinkcore) significantly.

[quintijn]

In principle, I support this idea, as you know. I hope, that this will make the install and maintenance procedure more simple. Starting a python process after starting Dragon is a minor extra thing when it makes other things more simple and robust.

OTOH, now the natlink subsystem is started with Dragon, and therefore integrated in the Dragon speech input cycle at the deepest possible place, and a supported way by Dragon.

I hope this close connection is not made weaker with your proposed solution. I think this needs to be discussed further!

[drmfinlay]

Thanks, Quintijn. I am confident that it will make installation, usage and maintenance simpler and more robust.

I appreciate your caution here, but I don't believe the connection will be made weaker by my proposed solution. I've used Natlink and Dragon this way extensively. The connection is the same one used by the test code — Natlink's and Dragonfly's.

If it does break, I take responsibility for fixing or reverting my changes.

On Sat, 06 Jul 2024 06:26:43 -0700 Quintijn Hoogenboom @.***> wrote:

In principle, I support this idea, as you know. I hope, that this will make the install and maintenance procedure more simple. Starting a python process after starting Dragon is a minor extra thing when it makes other things more simple and robust.

OTOH, now the natlink subsystem is started with Dragon, and therefore integrated in the Dragon speech input cycle at the deepest possible place, and a supported way by Dragon.

I hope this close connection is not made weaker with your proposed solution. I think this needs to be discussed further!

-- Reply to this email directly or view it on GitHub: https://github.com/dictation-toolbox/natlink/issues/198#issuecomment-2211768838 You are receiving this because you authored the thread.

Message ID: @.***>

[LexiconCode]

This will solve a number issues that would be difficult or impossible fix.

1. PYD registration is problematic. When Enabling natlink com object in Dragons ini files it is a global registration for all users. This causes an error for other users that may not be utilizing natlink with Dragon.

2. Admin privileges are required for registration pyd and editing dragen ini files.

3. If there's something wrong with natlink dragon has to be restarted. This is a sore spot for many users. Some are left helpless depending on their accessibility. Out of process means Dragon can be running in the background and the subsystem can be restarted with natlink. Then users can still control their computer.

At the end of the day, this makes Natlink more portable and robust for the end user.

There's one downside with how things currently would be structured out of process. Natlink does not start with Dragon automatically. This is a desired feature by user's.

On the process of making a GUI that can display command history, rules and so forth there might be another opportunity. My thought is this GUI could be running in the system tray when OS starts. When Dragon is launched, it could also be watching for that process and automatically start natlink.

The tricky bit is making sure it starts late enough. That might be worthy of some discussion..

[drmfinlay]

We agree on most of the above, I think. An autostart feature goes beyond what I had in mind here. As Quintijn said, it is a minor extra thing for the user to do. Maybe it can be considered further down the road? I am still in the planning phase on this. A polished GUI like that might be good as a setuptools "extra" for 'natlinkcore', to be installed alongside it.

I can help you with the last part at least. Use the `natlink.isNatSpeakRunning()' function.

[quintijn]

Great, Dane, I did not think about the isNatspeakRunning function.

Eventually, you only need to start Natlink: if Natspeak is not running yet it is started from the script, with configurable user (in natlink.ini), wait for the load procedure of Natstpeak. After that, you can go ahead.

[LexiconCode]

Here's my attempt to change how PYD gets loaded.

• Old way, registered COM object https://github.com/dictation-toolbox/natlink/blob/a48702f9f8992f3b12beb0f19f404b60676febf9/pythonsrc/src/natlink/__init__.py#L23-L73

• New way, finds PYD via path stored in environment.json instead of looking for registered COM object https://github.com/dictation-toolbox/natlink/blob/7bc3c1395c20ba27bab4fe91d1838bf27567e62b/pythonsrc/src/natlink/__init__.py#L32-L65

It seems the PYD is loaded successful but something still not right as evidenced by the following tested through dragonfly.

[24644] Loading C:\Users\Main\Desktop\natlink_vert_out_of_process\environment\DNS\venv\Lib\site-packages\natlink\_natlink_core.pyd from C:\Users\Main\Desktop\natlink_vert_out_of_process\environment\DNS\venv\lib\site-packages\natlink\__init__.py
[24644] RefCountedObject Create: ProfileInfoStatus addr 0x03F8AE48

========= Loading Dragonfly Example Rule ==========
INFO:engine:Initialized 'natlink' SR engine: NatlinkEngine().
DEBUG:command:Recognizing with engine 'natlink'
INFO:module:CommandModule('dragonfly_example_rule.py'): Loading module: 'C:\Users\Main\Desktop\natlink_vert_out\dragonfly_example_rule.py'
DEBUG:grammar.load:Grammar sample: adding rule MainRule.
DEBUG:grammar.load:Grammar sample: loading into engine NatlinkEngine().
DEBUG:grammar.load:Grammar sample: adding rule _IntegerRef_07.
DEBUG:engine:Engine NatlinkEngine(): loading grammar sample.
DEBUG:engine.compiler:NatlinkCompiler(): Compiling grammar sample.
DEBUG:engine.compiler:NatlinkCompiler(): Compiling rule MainRule.
DEBUG:engine.compiler:NatlinkCompiler(): Compiling rule _IntegerRef_07.
DEBUG:grammar.load:Grammar sample: activating rule MainRule.
DEBUG:engine:Activating rule MainRule in grammar sample.
DEBUG:grammar.load:Grammar sample: activating rule _IntegerRef_07.
DEBUG:grammar.load:Grammar _recobs_grammar: adding rule _anonrule_000_Rule.
DEBUG:grammar.load:Grammar _recobs_grammar: loading into engine NatlinkEngine().
DEBUG:engine:Engine NatlinkEngine(): loading grammar _recobs_grammar.
DEBUG:engine.compiler:NatlinkCompiler(): Compiling grammar _recobs_grammar.
DEBUG:engine.compiler:NatlinkCompiler(): Compiling rule _anonrule_000_Rule.
DEBUG:grammar.load:Grammar _recobs_grammar: activating rule _anonrule_000_Rule.
DEBUG:engine:Activating rule _anonrule_000_Rule in grammar _recobs_grammar.
Speech start detected.
DEBUG:grammar.begin:Grammar sample: detected beginning of utterance.
DEBUG:grammar.begin:Grammar sample: executable 'C:\Program Files\WindowsApps\Microsoft.WindowsNotepad_11.2405.13.0_x64__8wekyb3d8bbwe\Notepad\Notepad.exe', title 'Untitled - Notepad'.
DEBUG:grammar.begin:Grammar sample:     active rules: ['MainRule', '_IntegerRef_07']

See above after beginning of utterance grammar decode never happens above despite dragonfly showing an active rule!

Below is with the Dragonfly text engine

========= Loading Dragonfly Example Rule ==========
INFO:engine:Initialized 'text' SR engine: TextInputEngine().
DEBUG:command:Recognizing with engine 'text'
INFO:module:CommandModule('dragonfly_example_rule.py'): Loading module: 'C:\Users\Main\Desktop\natlink_vert_out\dragonfly_example_rule.py'
DEBUG:grammar.load:Grammar sample: adding rule MainRule.
DEBUG:grammar.load:Grammar sample: loading into engine TextInputEngine().
DEBUG:grammar.load:Grammar sample: adding rule _IntegerRef_07.
DEBUG:engine:Engine TextInputEngine(): loading grammar sample.
DEBUG:grammar.load:Grammar sample: activating rule MainRule.
DEBUG:grammar.load:Grammar sample: activating rule _IntegerRef_07.
hotel info
Speech start detected.
DEBUG:grammar.begin:Grammar sample: detected beginning of utterance.
DEBUG:grammar.begin:Grammar sample: executable 'C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.20.11381.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe', title 'C:\WINDOWS\system32\cmd.exe'.
DEBUG:grammar.begin:Grammar sample:     active rules: ['MainRule', '_IntegerRef_07'].
DEBUG:grammar.decode:   attempt: MainRule(MainRule)
DEBUG:grammar.decode:    -- Decoding State: ' >> hotel info'
DEBUG:grammar.decode:      attempt: Alternative(...)
DEBUG:grammar.decode:         attempt: Compound('hotel info')
DEBUG:grammar.decode:            attempt: Literal(['hotel', 'info'])
DEBUG:grammar.decode:            success: Literal(['hotel', 'info'])
DEBUG:grammar.decode:             -- Decoding State: 'hotel info >> '
DEBUG:grammar.decode:         success: Compound('hotel info')
DEBUG:grammar.decode:      success: Alternative(...)
DEBUG:grammar.decode:   success: MainRule(MainRule)
Recognized: hotel info
DEBUG:action.exec:Executing action: 'These types of hospitality services are not cheap.' ({'_grammar': Grammar(sample), '_rule': MainRule(MainRule), '_node': Node: Alternative(...), ['hotel', 'info'], 'n': 1, 'text': ''})
These types of hospitality services are not cheap.

It would be nice to get a simpler test just with natlink without dragonfly. I've sent through chat (It's too big to upload here) natlink_vert_out.zip which contains a virtual Python setup with the modified packages for testing purposes. make sure to unregister natlink before running natlink out of process.

One thing I'm not sure is how the pathway of natlink c++ is initialized out of process. For example relevancy of appsupp.h https://github.com/dictation-toolbox/natlink/blob/virtual_python/NatlinkSource/COM/appsupp.h

[dougransom]

alternative ways to get dragon out of process include DLL Surrigates https://learn.microsoft.com/en-us/windows/win32/com/using-the-system-supplied-surrogate.

You can use the Running Object Table to host a singleton natlink object (or a bunch of COM objects as you desire) that can accessed by name.

You can also host natlink in an NT Service if you really wanted to, so it is always available for natlink to connect to.

I have no idea if any of the above are applicable or helpful to what you are working towards.

[LexiconCode]

Here's a quick easy way to test that have been successful on my system. The issue of recognizing commands during the Decode time at least in dragonfly is an issue. I have yet to test directly with natlink. Thank you @quintijn for simplifying this process!

Download natlink_vert_out_of_process.zip link is valid for 30 days

1. Uninstall existing Natlink
2. extract natlink_vert_out_of_process.zip
3. Run setup.bat - Sets up everything
4. Running tests:
   • Run test_dragonfly - natlink.bat to test dragonfly with natlink or test_dragonfly - text.bat with dragonfly's Text engine without natlink. or
   • Execute python directly in python_environment.bat after typing python See Test Examples below:

Warning

• There are issues with leaving Natlink processes in a dirty state! On error probably not being able to execute natlink.natDisconnect()?

  • Workaround: Restart your computer but I'm looking for a better alternative like closing out certain processes. example: Python.exe/COM surrogate.exe

  The dirty state looks like the following in two ways:

  1. Dragon is running Test Example commands can freeze or or cause the GUI not to not respond in Dragon.
  2. Run Test Example when Dragon is completely close yet these commands in Test Example still think Dragon is running!

• Do not move the root folder as Python virtual environments rely on absolute paths unfortunately.

  • Workaround: Re-create the Python environment by deleting environment folder. Rerun setup.bat

Reverting to Natlink installer

1. Make sure all processes are closed, that Natlink is not in a dirty state and PYD is not in use.
2. run Natlink installer
3. Done

Technically I'm not sure if it's even required to uninstall/deregister Natlink. However I have run into issues where the virtual Python tries to pick up on natlink site packages in C:\Program Files (x86)\natlink. For testing purposes use a clean system without Natlink. No need to uninstall system Python.

Test Example

(venv) C:\Users\Main\Desktop\natlink_vert_out_of_process>python
Python 3.10.14 (main, Apr 15 2024, 17:42:09) [MSC v.1929 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import natlink
>>> import natlinkcore
>>> dir(natlink)
['BadGrammar', 'BadWindow', 'DataMissing', 'DictObj', 'GramObj', 'InvalidWord', 'MimicFailed', 'NatError', 'NatlinkConnector', 'OutOfRange', 'ResObj', 'SyntaxError', 'UnknownName', 'UserExists', 'ValueError', 'W32OutputDebugString', 'WrongState', 'WrongType', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_execScript', '_natlink_core', '_original_natconnect', '_playEvents', '_playString', '_recognitionMimic', '_test_playEvents', 'addWord', 'contextlib', 'createUser', 'ctypes', 'data', 'deleteWord', 'displayText', 'execScript', 'ext_keys', 'finishTraining', 'getAllUsers', 'getCallbackDepth', 'getClipboard', 'getCurrentModule', 'getCurrentUser', 'getCursorPos', 'getDNSVersion', 'getMicState', 'getScreenSize', 'getTrainingMode', 'getUserTraining', 'getWordInfo', 'getWordProns', 'get_config', 'importlib', 'inputFromFile', 'isNatSpeakRunning', 'json', 'lmap', 'loader', 'locale', 'natConnect', 'natDisconnect', 'openUser', 'os', 'outputDebugString', 'path_to_pyd', 'playEvents', 'playEvents16', 'playString', 'recognitionMimic', 'saveUser', 'setBeginCallback', 'setChangeCallback', 'setMicState', 'setTimerCallback', 'setTrayIcon', 'setWordInfo', 'spec', 'startTraining', 'toWindowsEncoding', 'traceback', 'waitForSpeech', 'win32api', 'win32gui', 'winreg', 'wrappedNatConnect']
>>> natlink.natConnect(1)
<contextlib._GeneratorContextManager object at 0x00B070A0>
>>> natlink.getMicState()
'on'
>>> natlink.setMicState('off')
>>> natlink.setMicState('on')

This will execute the loader manually in natlinkcore
>>> dir(natlinkcore)
['Path', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'getThisDir', 'logname']

>>> from natlinkcore import loader
>>> loader.run()
>>> natlink.natDisconnect()

I will be codifying these into tests that can be executed by bat files running python scripts. If the culprit is figured out by closing processes I can create dedicated bat to forcibly clean the state. That way we can test figure out if anything specifically contributing to this bad state.

I will make a new post with the new zip file when these tests are ready! Currently utilize what's above.

[LexiconCode]

I've tried registering pyd without my changes natlink init and leaving the Dragon ini vanilla. Dragonfly called through the cli with a demo grammar (Although I don't see decode in debug mode) are recognized, but not executed. The same, calling through natlink!

[drmfinlay]

I have not had a chance to test this yet, but these results do not sound encouraging. I suspect it is a problem with Dragon 16. I'll look into it and see what, if anything, can be done.

On Sat, 13 Jul 2024 21:55:40 -0700 LexiconCode @.***> wrote:

I've tried registering pyd without my changes natlink init and leaving the Dragon ini vanilla. Dragonfly called through the cli with a demo grammar dommands ( Although I don't see the decode debug mode) are recognized, but not executed. The same, calling through nalink!

-- Reply to this email directly or view it on GitHub: https://github.com/dictation-toolbox/natlink/issues/198#issuecomment-2227195836 You are receiving this because you authored the thread.

Message ID: @.***>

[LexiconCode]

natlink_vert_out_of_process_v2

• Added more bat files to run .py file for tests

[drmfinlay]

The issue mentioned above — the broken recognition callback — appears to be Windows-related. I can reproduce it on Windows 10 with Dragon 15. It does not occur on Windows 7 with the same Dragon version.

The problem may be fixable. I have been able to determine that the callback is invoked, but causes a fault. I'll see if anything can be done about that.

I don't know about the dirty state issue. It does sound like what occurs when `natDisconnect()' isn't called.

[drmfinlay]

Okay, good news. I managed to fix the broken callback. My proposal now looks viable on Windows 10. The same binary also works on Windows 7. Dragonfly's test suite now passes expectedly on both.

I did have to do some debugging and I can confirm it is easier this way. Simply attach Visual Studio's native debugger to your Python process and ensure the symbol file (natlink.pdb) is loaded.

[drmfinlay]

@LexiconCode The dirty state issue that occurs if one closes Dragon before Natlink is solvable by waiting for the shutdown event and disconnecting cleanly. I have some code for this that works and will include it in my changes for review.

I assume you guys would like Natlink to remain open if Dragon is closed first and then reconnect when it can, rerunning the loader code. This seems intuitive to me.

[LexiconCode]

Yes, that seems intuitive to me. Is it a simple, wait for dragen.exe to load then execute loader?

[drmfinlay]

Okay great. Yes, basically. There is a Windows event for DNS initialization that the program can wait for.

[dougransom]

I think it is ok to make the user launch natlink, or to launch it at startup, or when dragon restarts (which is not that often).