Is it possible to finetune the MMS model?

[CopyNinja1999]

Is it possible to finetune the MMS model? I didn't find any doc or code supporting this feature.

[patrickvonplaten]

We're working on making this very easy in transformers. You can check:

• https://huggingface.co/docs/transformers/main/en/model_doc/mms
• https://github.com/huggingface/transformers/pull/23813
• https://huggingface.co/facebook/mms-1b-all

Stay tuned!

[CopyNinja1999]

@patrickvonplaten Hi, could you please also upload the pretrained discriminator for mms-tts models? Also, I met this error when fine-tuning the Korean tts model using random discriminator checkpoint: packages/bitsandbytes/optim/optimizer.py", line 455, in update_step if state["state1"].dtype == torch.float: KeyError: 'state1'

[allandclive]

https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#connectionist-temporal-classification-with-adapters

[patrickvonplaten]

Also a more step-by-step blog post here: https://huggingface.co/blog/mms_adapters