Open josancamon19 opened 1 month ago
Hello, I'll gladly take this issue. My plan is:
Integrate VAD: I will incorporate the 'silero-vad' library, which is well suited for Friend device, for better voice activity detection.
Adjust Audio Buffer Handling: I'll refine the handling of audio data, managing the buffer size and ensuring that it correctly handle different audio formats, such as Opus and PCM.
Sample Rate and Codec Handling: I'll try to involve adjusting the VAD parameters and buffer calculations based on the specified sample rate and codec.
Looking forward for reply fren :)
Awesome! assigning to @0xzre for the next 2 days.
Some context of what is in place already: https://github.com/BasedHardware/Friend/blob/main/backend/utils/stt/vad.py https://github.com/BasedHardware/Friend/blob/272b663b0d86832e56a0ccea3656b7f372e8361a/backend/routers/transcribe.py#L66
Hi @0xzre can you submit a Draft PR and show progress?
Fixed for pcm8 and pcm16. Opus is still pending
Is your feature request related to a problem? Please describe. VAD needs to determine better when to send or not to send bytes.
File
transcribe.py
/listen endpoint.Describe the solution you'd like Opus 16k 16 bit. pcm8 for old firmware version. 8khz. pcm16 for from device recording.
This requires also to work with multiple languages.