search

daanzu / kaldi-active-grammar

Python Kaldi speech recognition with grammars that can be set active/inactive dynamically at decode-time
GNU Affero General Public License v3.0
332 stars 49 forks source link

Portaudio related crashes with kag-v1.8.0 #35

Closed JohnDoe02 closed 3 years ago

JohnDoe02 commented 3 years ago

Today, I pulled the update to kag-1.8.0 and downloaded the new large language model. After upgrade of kaldi-active-grammar I ran python setup.py bdist_wheel to ensure that kaldi is rebuild as well and of course pulled the updates for dragonfly.

However, after a few voice commands I am experiencing crashes:

Speech start detected.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
Traceback (most recent call last):
  File "kaldi_module_loader_plus.py", line 180, in <module>
    self._target(*self._args, **self._kwargs)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/backend_kaldi/audio.py", line 100, in _reader_thread
    main()
    in_data, overflowed = self.stream.read(self.stream.blocksize)
  File "kaldi_module_loader_plus.py", line 174, in main
  File "/home/user/versioned/voice-coding/env/lib/python3.8/site-packages/sounddevice.py", line 1196, in read
    engine.do_recognition(on_begin, on_recognition, on_failure)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/base/engine.py", line 260, in do_recognition
    _check(err)
  File "/home/user/versioned/voice-coding/env/lib/python3.8/site-packages/sounddevice.py", line 2653, in _check
    self._do_recognition(*args, **kwargs)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/backend_kaldi/engine.py", line 372, in _do_recognition
    raise PortAudioError(errormsg, err)
    kaldi_rule, words, words_are_dictation_mask, in_dictation = self._compiler.parse_partial_output(output)
sounddevice.PortAudioError: Stream is stopped [PaErrorCode -9983]
  File "/home/user/versioned/voice-coding/kaldi-active-grammar/kaldi_active_grammar/compiler.py", line 573, in parse_partial_output
    assert nonterm_token.startswith('#nonterm:rule')
AssertionError
JohnDoe02 commented 3 years ago

Just to make sure, I just retried with the small model. Same problem though. The error message was slightly different this time:

Speech start detected.
Traceback (most recent call last):
  File "kaldi_module_loader_plus.py", line 180, in <module>
    main()
  File "kaldi_module_loader_plus.py", line 174, in main
    engine.do_recognition(on_begin, on_recognition, on_failure)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/base/engine.py", line 260, in do_recognition
    self._do_recognition(*args, **kwargs)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/backend_kaldi/engine.py", line 372, in _do_recognition
    kaldi_rule, words, words_are_dictation_mask, in_dictation = self._compiler.parse_partial_output(output)
  File "/home/user/versioned/voice-coding/kaldi-active-grammar/kaldi_active_grammar/compiler.py", line 573, in parse_partial_output
    assert nonterm_token.startswith('#nonterm:rule')
AssertionError
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/versioned/voice-coding/dragonfly/dragonfly/engines/backend_kaldi/audio.py", line 99, in _reader_thread
    if self.stream.active and self.stream.read_available >= self.stream.blocksize:
  File "/home/user/versioned/voice-coding/env/lib/python3.8/site-packages/sounddevice.py", line 981, in active
    return _check(_lib.Pa_IsStreamActive(self._ptr)) == 1
  File "/home/user/versioned/voice-coding/env/lib/python3.8/site-packages/sounddevice.py", line 2653, in _check
    raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Invalid stream pointer [PaErrorCode -9988]
python kaldi_module_loader_plus.py  41.86s user 12.57s system 85% cpu 1:03.57 total

Seemingly, there is some problem in the interaction with portaudio?

JohnDoe02 commented 3 years ago

I reverted back to kag-v1.7.0. As expected, everything is working fine again with this version.

daanzu commented 3 years ago

Curious. I will investigate. Does v1.8.0 work using the pip package (python -m pip install -U kaldi-active-grammar==1.8.0)? I think the portaudio error may be a red herring, only occurring because the other error having already occurred.

JohnDoe02 commented 3 years ago

Same problem with the pip package. I installed it into a fresh python env, together with pip install dragonfly2[kaldi] and my grammar.

Timoses commented 3 years ago

I seem to be hitting the same assertion.

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/__main__.py", line 408, in <module>
    main()
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/__main__.py", line 403, in main
    return_code = func(args)
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/__main__.py", line 181, in cli_cmd_load
    _do_recognition(engine, args)
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/__main__.py", line 96, in _do_recognition
    engine.do_recognition()
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/engines/base/engine.py", line 260, in do_recognition
    self._do_recognition(*args, **kwargs)
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/dragonfly/engines/backend_kaldi/engine.py", line 372, in _do_recognition
    kaldi_rule, words, words_are_dictation_mask, in_dictation = self._compiler.parse_partial_output(output)
  File "/Users/Timoses/code/voice/caster/venv/lib/python3.8/site-packages/kaldi_active_grammar/compiler.py", line 573, in parse_partial_output
    assert nonterm_token.startswith('#nonterm:rule')
AssertionError

i am using the big KAG model from 1.8.0 together with the same version of KAG. Also using the newer version of dragonfly (0.27.0).

daanzu commented 3 years ago

This should be fixed in the just released v1.8.1. Please try upgrading (pip install -U kaldi-active-grammar) and let me know.

Timoses commented 3 years ago

Seems to be working for me. Great job!