To make Trump speak Chinese

panshaoning commented 3 weeks ago

Self Checks

[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

To make Trump speak Chinese, I fine-tuned a model based on Trump's English dataset and used the tools/llama/merge_lora.py script to convert the LoRA weights to regular weights. However, when I perform inference with the new model, the generated speech is always in English and not in Chinese. I'm not sure why this is happening.

2. Additional context or comments

No response

3. Can you help us with this feature?

[X] I am interested in contributing to this feature.

PoTaTo-Mika commented 1 week ago

We recommend you to use Trump's voice as reference and directly generating voice with Chinese text.

HuuHuy227 commented 1 week ago

We recommend you to use Trump's voice as reference and directly generating voice with Chinese text.

Is it the problem of catastrophic forgetting when finetuning? I'm facing the same issue when trying finetune on Vietnamese dataset and after merged model I use an unseen voice as reference but generated speech always say the same style like in my dataset

fishaudio / fish-speech