search

matatonic / openedai-speech

An OpenAI API compatible text to speech server using Coqui AI's xtts_v2 and/or piper tts as the backend.
GNU Affero General Public License v3.0
457 stars 59 forks source link

Cuda support for piper #51

Open thiswillbeyourgithub opened 2 months ago

thiswillbeyourgithub commented 2 months ago

Hi, i saw that in this commit you removed on purpose the cuda support for piper because apparently it didn't work for your setup?

I did a few test and it seemed to worked fine for me and was wondering if you could consider re adding support for it.

For me it was as simple as pip install onnxruntime-gpu then add --piper to the subprocess call. It could stay opt in and need an argument in the voice yaml for example.

What do you think? Am I missing something as to why it should be removed? Is it the size of the image?

Thanks!

matatonic commented 2 months ago

Interesting, yeah it was removed because it didn't work in the docker image. Maybe it's fixed! I'll check into adding onnxruntime-gpu back into the image.

thiswillbeyourgithub commented 1 month ago

No update on that front by any chance?

matatonic commented 1 month ago

not yet.

thiswillbeyourgithub commented 1 month ago

Fyi after taking a look at the cuda issues in the piper repo I think it would be a much better bet for low end users to try to setup the fish model with quantization ( #58 ).

Btw just found out about your other opened ai projects. Really awesome stuff thank you very much! Do you accept donations by any chance? I wanted to offer you a few beers coffees in advance while showing what feature would help me the most (in that case fisher support). Kinda like a bounty but with no strings attached of course. Have a nice day!

matatonic commented 1 month ago

I probably wont setup any tip thing, but thanks for the kind words. If you're ever Toronto, I'd happily accept the drinks in person :-)

I know I'm lagging a lot updating speech, but I expect to get around to it again soon!

thetznecker commented 1 month ago

@thiswillbeyourgithub did you see any performance gains while using gpu? CPU generation takes me around 0.5-0.7 seconds on a i9-14900kf, while gpu takes a couple seconds on a 3090.

thiswillbeyourgithub commented 1 month ago

I probably wont setup any tip thing, but thanks for the kind words. If you're ever Toronto, I'd happily accept the drinks in person :-)

Nice! Unfortunately I'm in thr capital of Baguette land so if you come by your drink is on me :)

@thiswillbeyourgithub did you see any performance gains while using gpu? CPU generation takes me around 0.5-0.7 seconds on a i9-14900kf, while gpu takes a couple seconds on a 3090.

I have not succedded in getting cuda to work on my piper setup in less than 20 minutes and it was about the time I had on my hand so can't help sorry.