Closed bjsi closed 10 months ago
Hey very cool, this is something I was thinking about adding. Is it for full duplex text-to-speech? It would be the first if its kind in ModelFusion, so I'd prefer to implement it myself, but would love to chat.
Yeah basically as tokens are streamed from GPT they get sent to ElevenLabs via websocket so you don't have to wait for GPT to finish generating before audio generation starts.
I've started working on this in #144 - feel free to check it out (it's just a prototype for now that works in node.js, not tested in the browser).
Awesome I'll check it out later
On Wed, 18 Oct 2023, 09:42 Lars Grammel, @.***> wrote:
I've started working on this in #144 https://github.com/lgrammel/modelfusion/pull/144 - feel free to check it out (it's just a prototype for now that works in node.js, not tested in the browser).
— Reply to this email directly, view it on GitHub https://github.com/lgrammel/modelfusion/issues/141#issuecomment-1767972306, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN3UCAZRODJX67O37YIDDPDX76JAXAVCNFSM6AAAAAA6A2K66CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRXHE3TEMZQGY . You are receiving this because you authored the thread.Message ID: @.***>
Implement for most parts, see https://github.com/lgrammel/modelfusion/tree/v0.48.0/examples/duplex-speech-streaming-vite-react-fastify for a more complex example.
I still need to add observability, some ElevenLabs API parameters, and base64 forwarding.
I got this set up for a project over the weekend and it was such a pain 😅 it would be nice to have it wrapped up in a library like modelfusion.
Here's the code I ended up with for the browser:
I'd be glad to implement this into modelfusion myself and make a PR if you think it's a reasonable addition to the library (I might need some guidance). I'd like to support both web and browser environments if possible. Let me know what you think :)