Add HTTP streaming for local models

[Cohee1207]

Description

Adds a proper TTS streaming via HTTP by using coqui's inference_stream method and fastapi's StreamingResponse. The client can consume the new data as soon as it's ready. I found the chunk size of 100 (coqui tokens I assume?) to provide a favorable latency/interruption rate on my MacBook running CPU inference.

Implications

1. Works only with local models.
2. Uses HTTP GET instead of HTTP POST. Explanation below.

Initially, I wanted to stick to HTTP POST requests only and do audio playback using client-side JavaScript, but unfortunately MediaSource does not support working with WAV data. Adding intermediate compression would only increase latency and create more complexity. Using HTTP GET allows doing playback directly from HTML by setting the audio source to the API endpoint, the browser will do all the buffering and decoding at no extra cost.

Related SillyTavern pull request: https://github.com/SillyTavern/SillyTavern/pull/1623

References

• https://github.com/erew123/alltalk_tts/pull/36
• https://docs.coqui.ai/en/latest/models/xtts.html#streaming-manually
• https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

[daswer123]

Wow great job, I'll check it out now

[daswer123]

Checked everything works great, thanks for your work :)