search

QuantumEntangledAndy / neolink

An RTSP bridge to Reolink IP cameras
GNU Affero General Public License v3.0
250 stars 41 forks source link

Attempts to fix talk stability with audio files #199

Closed Frans-Willem closed 3 months ago

Frans-Willem commented 7 months ago

My goal is to be able to send text-to-speech messages from home assistant to my Reolink Doorbell POE. The current neolink version has issues for me where the message is cut off too soon, and with longer messages, the neolink executable would never exit. The never exiting seems to be because the response to the TALK command is not handled, which is tackled in the first commit. This made the playback choppy, as the thread::sleep after each audio chunk was no longer correct. I've attempted to fix this by keeping track of when the message should end, and sleeping until just before that before sending a new chunk. The cutting off of the message early was also fixed by sleeping until just after the supposed message end.

I believe the message_end calculation should work well for audio files, but might start drifting with streamed audio (e.g. microphone). I'm not quite sure how I'd properly tackle that, but I'm open to suggestions. I also think the talk and talk_stream functions should probably be merged, such that fixes in one do not have to be repeated in the other.

TLDR; Let's have a discussion on whether these are the required fixes or if something else needs work.

Frans-Willem commented 7 months ago

I've tested this using: ./target/debug/neolink talk doorbell --config ../neolink_config.toml -m -i "souphttpsrc location=https://cdn.openai.com/API/docs/audio/alloy.wav"

Frans-Willem commented 7 months ago

More fixes, turns out sleeping between or timing the sending of chunks is not needed if you just wait for the response, the doorbell will slow the responses down if the buffer is full. I've first attempted to estimate the buffer length, but I couldn't get it fully reliable. Instead, I keep track of how much streaming data was sent in seconds, and use that to sleep a little before doing talk_stop. Seems to work OK, and I expect the current method to also work for streaming a microphone.

QuantumEntangledAndy commented 3 months ago

Thanks. Sorry for the delay. Got a new job stuff and it's been very busy. Hopefully this will help with the choppy audio