search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
29.86k stars 6.32k forks source link

Is same base model used for MMS-ASR and MMS-TTS (like AudioPaLM)? #5221

Open kippapollo opened 1 year ago

kippapollo commented 1 year ago

Google introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation.

How about the MMS? I found fine-tuning MMS ASR based on pretrained base model mms-1b, but I can not find for TTS. Is the same base model mms-1b used for MMS-TTS? How can I fine-tuning or add new language for TTS?

CopyNinja1999 commented 1 year ago

TTS models are separate. They are all vits-based models so imagine fine-tuning vits checkpoint