Open zclch opened 7 months ago
This is a nice idea. The trick seems to be getting good ground truth emotional speech samples and labels that don't sound fake.
This is not currently supported in any way but there are people in the LAION Discord working on emotional speech so I'll invite you to join the #audio-generation there channel if you are interested.
This is a nice idea. The trick seems to be getting good ground truth emotional speech samples and labels that don't sound fake.
This is not currently supported in any way but there are people in the LAION Discord working on emotional speech so I'll invite you to join the #audio-generation there channel if you are interested.
@jpc , that sounds great, I would love to follow along with their progress. How can we get in contact?
It’s best if you ask on the LAION Discord (link in the README).
If I can collect a large dataset of clean speech, annotated with text and "emotion vectors" which I assume to be accurate, how can I train whisperspeech on it? This would be like training a new multimodal tts model, going from text + emotion vectors -> speech. Can you give me high level guidance on where in the training pipeline I could add support for this extra modality and how to do it?
It would be amazing if emotion markers can be supported (or if they already are, documentation on how to use them), for example providing indicators like
<angry>
,<excited>
, etc. or use of emoji's for the same.