search

mkiol / dsnote

Speech Note Linux app. Note taking, reading and translating with offline Speech to Text, Text to Speech and Machine translation.
Mozilla Public License 2.0
389 stars 19 forks source link

Enhancement: Parler-TTS, a new open source model with steerable voice characteristics #122

Open phirsch opened 2 months ago

phirsch commented 2 months ago

Wondering whether this could be an interesting model to add - it allows the voice characteristics to be altered using an auxiliary prompt.

Quote from https://github.com/huggingface/parler-tts:

Parler-TTS is a lightweight text-to-speech (TTS) model that can generate high-quality, natural sounding speech in the style of a given speaker (gender, pitch, speaking style, etc). It is a reproduction of work from the paper Natural language guidance of high-fidelity text-to-speech with synthetic annotations by Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively. Contrarily to other TTS models, Parler-TTS is a fully open-source release. All of the datasets, pre-processing, training code and weights are released publicly under permissive license, enabling the community to build on our work and develop their own powerful TTS models.

P.S.: Thanks for this great software - and extra kudos for the modular GPU acceleration FlatPak release!

mkiol commented 2 months ago

Thanks for letting know about this model. This creation of voice characteristics using text prompt is very interesting and unique! Parler-TTS would be a great addition to Speech Note.

The only problem is that this model cannot generate long text. Therefore, you need split the text into sentences and generate each sentence separately. Unfortunately, this causes each sentence to be generated in a different voice because this problem occurs.

I'm waiting for Parler-TTS devs to fix this problem and then I will definitely integrate it into Speech Note.

phirsch commented 2 months ago

Valid points there - thanks for considering the model and finding this out!

I agree that switching voices between any two consecutive sentences is pretty awkward (IIRC, some included models - maybe WhisperSpeech? - already exhibit this to some extent) - particularly for my main use case (helping to quickly but still accurately skim scientific papers).

Feel free to close the issue unless you prefer to keep it open as a reminder.