search

AndrewVeee / nucleo-ai

An AI assistant beyond the chat box.
Apache License 2.0
313 stars 29 forks source link

Add speech recognition #9

Open harish0201 opened 7 months ago

harish0201 commented 7 months ago

This is an idea that just popped up, while I was trying to search something.

Maybe we can integrate whisper.cpp or something like that, for interaction? It would be awesome if the Assistant can search the web then, and pass it through the summarization engine.

AndrewVeee commented 7 months ago

Sorry for the delay in responding - had a little vacation after the release.

I think it would be great to test this out using the API instead of building it into the app from the start. I've seen a few posts on r/LocalLlama about using whisper with LLMs, so it doesn't seem too difficult. You would just have to read in the whisper speech-to-text, then using the openai lib with model="assistant", messages=[{"role": "user", "content": whisper_stt}]), and it should do its thing.

Of course, it would also be great to have tts as well.