Issues with interrogative sentences when using ljspeech model

shivammehta25 / Matcha-TTS

[ICASSP 2024] 🍵 Matcha-TTS: A fast TTS architecture with conditional flow matching

https://shivammehta25.github.io/Matcha-TTS/

MIT License

716 stars 87 forks source link

Issues with interrogative sentences when using ljspeech model #79

Closed orenstuf closed 4 months ago

orenstuf commented 4 months ago

Hello,

Here are some issues with interrogative sentences:

"May I take your order?" - sounds like a standard declarative sentence.
"Where do you live?" - sounds like a standard declarative sentence and adds stress on the word "you".
"Did you finish your homework on time?" - sounds like a question, but the intonation in incorrect.

Can you give me some guidelines on how to train matcha to overcome these issues?

Thanks a lot

shivammehta25 commented 4 months ago

Hi @orenstuf,

The LJ Speech checkpoint is only trained on LJ Speech corpus. Being a read speech corpus it doesn't really have a lot of variation or interrogative sentences for the model to learn.

The only mitigation for this will be to improve the dataset. Or try fine-tuning the model to your voice. Record 20 such sentences and fine-tune and see if the model learns something from it.

Regards, Shivam