Commands using Kaldi engine don't seem to work anymore or only work in specific ways

[jdcai]

Describe the bug Some commands don't seem to work anymore or only work in specific ways when using Caster with the Kaldi engine

Some of the ones I found so far: switch window sometimes works or sometimes does nothing in chrome go back doesn't work unless you say go back n where n is a number greater than 1. It will do the command n-1 times. Ex: If you say go back 2 it will go back 1. Same thing happens with others that use n like go forward n, back tab n, next tab n also happens with shock which doesn't work unless you say shock 2.

Seems like commands that use a function still work like kick, or commands that have %(n)s like latch's R(Key("home:%(nnavi10)s")). Commands with * Repeat(extra="n") work if you say 2 or higher. If its just a simple one like R(Key("t") then it doesn't seem to do anything.

System:

• OS: Windows 10
• Program: Kaldi engine with Caster
• Version: master branch of Caster

[LexiconCode]

Carry over from the previous issue. Some of this information will turn into documentation as well. Some things keep in mind:

• Kaldi performs better when grammars are less ambiguous. So you may need customize as needed based on your experience with certain commands.

• Utilize some grammar debugging tools python -m dragonfly test _caster.py this takes the speech recognition out of the stack. Allows you to type words as if it spoken recognition. Lowercase words emulates commands, uppercase words act as free dictation which can be combined.

  • For example typeing gerrish snake VARIABLE NAME then variable_Name should type out after pressing enter.
  • You can utilize --delay flag with being the delay before the command is executed so you can switch the context needed. There are a few other options. https://dragonfly2.readthedocs.io/en/latest/cli.html#test-exampleshttps://dragonfly2.readthedocs.io/en/latest/cli.html#test-examples

• You can load any dragonfly rule via command line. For example sample.py as a vanilla dragonfly rule. python -m dragonfly load sample.py --engine kaldi --no-recobs-messages

  • May be useful for creating a test case that we can submit back to daanzu if we find issues with kaldi that is reproducible.

[daanzu]

I think this may be a case of https://github.com/dictation-toolbox/dragonfly/issues/182. Can you try enabling the logging and see if it fixes it?

[jdcai]

how do I enable the logging?

[LexiconCode]

Try python -m dragonfly load _*.py -l ERROR --engine kaldi

[jdcai]

Tried it and it still has the same problems. I noticed that the first command said after starting caster works properly. If I say a command like shock it works the first time. Afterwards saying it again it does nothing. I need to say shock 2 for it to do a enter.

[LexiconCode]

Tried it and it still has the same problems. I've created a test grammar. TestSample.zip Extract and drop in your Caster directory. From the same location in CMD run the following command.

python -m dragonfly load TestSample.py -l ERROR --engine kaldi

[jdcai]

It works fine using that. Tried it without -l ERROR and it worked fine too.

[daanzu]

Does that command line only log ERROR messages? Try logging at the DEBUG level, to test if this is a case of that other issue.

[LexiconCode]

I've tested this under Python 3 across two different computers without issue.

Does that command line only log ERROR messages?

Yes, Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

-q, --quiet Equivalent to -l WARNING – suppresses INFO and DEBUG logging.

It works fine using that. Tried it without -l ERROR and it worked fine too.

Try python -m dragonfly load _caster.py -l DEBUG --engine kaldi

[jdcai]

it works if I use -l DEBUG

[daanzu]

Thanks, I will try to look into it more. Very puzzling.

[daanzu]

This may have been fixed in dragonfly2 v0.22.0. Please let me know if you still need the debugging logging or not after upgrading to that version.

[jdcai]

Upgraded but unfortunately the problem still exists.

[daanzu]

Shucks. I assume enabling the logging still fixes it?

Unfortunately, I can't seem to reproduce it currently.

[drmfinlay]

@daanzu Are you still able to reproduce it in earlier dragonfly2 versions, e.g. 0.21.0?

Since I have never been able to reproduce this bug, I was wondering if you could track down which commit introduced it with git-bisect?

[daanzu]

@Danesprite A good idea, but nope. I have tried going back to all released versions back to november, and none of them seem to reproduce the problem.

I think it must be some strange timing issue, and now my computer is no longer demonstrating it.

[drmfinlay]

@daanzu Oh okay then. That is unfortunate, but thanks for trying anyway. Perhaps your logging fix should be added into Dragonfly anyway? It seems to have worked for a few people now.

[daanzu]

@Danesprite Yeah, I had wondered about adding something to automatically turn on the debug logging for actions if it is running on Kaldi on Windows. But then when I went to implement it, I discovered my inability to reproduce it. It is probably still a good idea to do so.

I will keep an eye out for it re-emerging, perhaps after I reboot (my current uptime is >100 days).

[drmfinlay]

@daanzu Sounds like a plan. I would be happy to merge a Dragonfly PR with this change. I think there needs to be a separate function for checking the current engine without initialising one as a side-effect.

my current uptime is >100 days

Impressive! I'm a little surprised Windows is still functional after running for so long!

[daanzu]

@daanzu Sounds like a plan. I would be happy to merge a Dragonfly PR with this change. I think there needs to be a separate function for checking the current engine without initialising one as a side-effect.

I am not quite sure where to put the code for enabling the logging.

For checking the current engine, I just do something like:

engine_kaldi = (engines.get_engine()._name == 'kaldi')

We should probably have a less hacky and more documented way of doing it.

my current uptime is >100 days

Impressive! I'm a little surprised Windows is still functional after running for so long!

Ha, yeah. I wonder if everything is running slower enough to mask the bug, instead of needing the logging to delay things.

[drmfinlay]

Regarding where to put the code for enabling the logging, how about in KaldiEngine.__init__()?

I could add a get_current_engine() function that returns None if one hasn't been initialized yet.

Ha, yeah. I wonder if everything is running slower enough to mask the bug, instead of needing the logging to delay things.

Ha, I suppose this is possible.

[LexiconCode]

A report from cmvanb on Windows 10 via Discord.

so far the recognition seems good, but I ran into some issues with keys like enter & shift shock is recognized as enter, but doesn't seem to get sent to windows 10.

Procedures:

• Verified that that commands work appropriately with the dragonfly test engine
• Ran Kaldi with debugging and the keys worked as expected.

Doesn't tell us much new atm but another person to test out potential fixes in the future.

[daanzu]

@LexiconCode Thanks for the report. I will put in something to automatically turn on the logging so this doesn't keep affecting people, until I can reproduce and then hopefully fix it.

[daanzu]

I added a temporary mitigation to enable logging in https://github.com/dictation-toolbox/dragonfly/pull/235 that was released in dragonfly2 v0.23.1. Or does Caster use setup_log()?

[LexiconCode]

Caster an upcoming pull request will be using

    _logger = logging.getLogger('caster')
    _logger.addHandler(LoggingHandler())  # must be after nexus initialization
    _logger.setLevel(logging.DEBUG)

Currently Just logging.basicConfig()

[drmfinlay]

@LexiconCode The problem should be fixed in v0.23.1 either way then. Dragonfly's setup_log() function needs to be adjusted for the temporary mitigation to work with it.

[LexiconCode]

Closing for now and will reopen if further reports materialize. Thanks everyone!