search

alexpinel / Dot

Text-To-Speech, RAG, and LLMs. All local!
https://dotapp.uk/
GNU General Public License v3.0
1.28k stars 60 forks source link

Voice Input button is stuck #16

Open couldbeme opened 2 months ago

couldbeme commented 2 months ago

Hey, thanks for your great work here! I was able to set up the project on M1 Mac (Sonoma), but unfortunately STT doesn't seem to be toggled.

Any possible solutions? Warmly, Alexa

alexpinel commented 2 months ago

Hi!

Sorry for the inconvenience, I am still working on updating the README to include all details regarding the voice input and TTS. The issue is most likely happening because whisper.cpp is not installed, to setup whisper.cpp you can follow the instructions here. Once done you should copy all the contents of the whisper.cpp folder into a folder called whisper and place it inside the llm directory.

Alternatively, if you have downloaded the Dot app you can right click to show package contents, from there you can navigate to contents/resources/llm and just copy paste the existing whisper folder into your project's llm folder. I do not know 100% if that would work though 😅

Also, to make the Text-to-speech work you might have to download the TTS model and place the downloaded contents inside /llm/vits-piper-en_US-glados, if prompted you can replace any duplicate files. The models are available here, I used the GLaDOS voice but any other should also work fine! You will have to change some of the file names in ttsProcessor.js if you choose another model.

Hope that was helpful! And please let me know if you are facing any other issues :)

Best,

Alex

couldbeme commented 2 months ago

Hi Alex, thank you very much for your fast response!

UPD: I have installed Dot application - and following package contents/resources/llm I can confirm that Whisper is installed, but ASR still doesn't work. I also have tried to copy whisper contents into Dot-Data/llm/whisper directory to try it out, but it didn't work either.

Will try to compile from source and update later whether it helped, maybe that will resolve the issue somehow. Rather than that, will much appreciate any possible options to check.

Warmly, Alexa

alexpinel commented 2 months ago

Hi Alexa, sorry to hear you are still facing the same issue!

The llm folder containing whisper should be placed in the same directory as the project itself alongisde src and node_modules, sorry if I did not make that clear. Also make sure to download the ggml model for whisper, Dot is configured to use ggml-model-whisper-base.bin, but you can use another one as long as you modify the name in line 764 of index.js. I have also just pushed a commit adjusting some of the paths to "dev" mode as they were pointing to the resources folder that would exist once the app is packaged.

I would also like to ask if the ASR worked for you on the packaged app, and if not could you please let me know if there were any error logs? Those should appear in the dev window by clicking Command+Shift+I. I have not found very good implementations of Whisper for node.js so I decided to use the original C++ binaries instead, but that could raise some compatibility issues between different machines.

Best regards,

Alex