kalliope-project / kalliope

Kalliope is a framework that will help you to create your own personal assistant.
https://kalliope-project.github.io/
GNU General Public License v3.0
1.71k stars 229 forks source link

Whisper support #686

Closed jaggzh closed 1 year ago

jaggzh commented 1 year ago

OpenAI-Whisper is a really high quality locally-run speech-to-text project. (It can also translate; but nevermind that for now).

It would be awesome if we could get that in here, even for the trigger word recognition. I started a separate project to use it for live transcription. Whisper runs as a single command, or from python. My project includes a little python flask app that keeps the model loaded, takes a wav file (POSTed), and returns just the text. https://github.com/jaggzh/whisperpluck

Before I do any effort to implement it in Kalliope I wanted to run it by you all. (I'm quite busy so I don't think I'll even be able to get to it for now. I coded whisperpluck for a friend who has issues using the computer).

joshuaboniface commented 1 year ago

687 adds this!

joshuaboniface commented 1 year ago

687 was merged, so the next release should have (core) Whisper support.

Next step is FasterWhisper, which is indeed much faster. This is waiting on the upstream PR https://github.com/Uberi/speech_recognition/pull/693 to implement in speech_recognition, but otherwise I have the code ready. If you want to test those out, the branch blkal of both Kalliope and Speech Recognition in my respective forks have my latest code for both.

jaggzh commented 1 year ago
I'm listening
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/jaggz/venvs/kalliope/lib/python3.11/site-packages/kalliope/core/OrderListener.py", line 52, in run
    self.stt_instance = self.load_stt_plugin()
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jaggz/venvs/kalliope/lib/python3.11/site-packages/kalliope/core/OrderListener.py", line 67, in load_stt_plugin
    return Utils.get_dynamic_class_instantiation(package_name='stt',
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jaggz/venvs/kalliope/lib/python3.11/site-packages/kalliope/core/Utils/Utils.py", line 135, in get_dynamic_class_instantiation
    mod = __import__(package_path, fromlist=[module_name.capitalize()])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'kalliope.stt.whisper'
^C

Ctrl+C pressed. Killing Kalliope
jaggzh commented 1 year ago

Oh, this happens after I say Kaliope (during STT?)

jaggzh commented 1 year ago

Oh, sorry, I was in master; I noticed in your PR it's in dev. That's working. :)) Closing this issue.