search

keonlee9420 / Cross-Speaker-Emotion-Transfer

PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
MIT License
179 stars 26 forks source link

Synthesis with other person out of RAVDESS #11

Open hathubkhn opened 2 years ago

hathubkhn commented 2 years ago

Hello author, Firstly, thank you for giving this repo, it is really nice. I have a question that:

  1. I download CMU data with single person with 100 audios and make speaker embedding vector and synthesis with this, the performance is not good. I cannot detect any words.
  2. Should we need to fine-tuning deep-speaker model to generate speaker embedding with my data.

Thank you

keonlee9420 commented 2 years ago

Hi @hathubkhn , thanks for your attention.

  1. There could be a various reason for such case. Could you please share the tensorboard logs and some samples audio with mel-spectrogram?
  2. It might be, but it depends on the number of speakers and their features.
hathubkhn commented 2 years ago

Hi,

  1. When waiting for your response, I try to finetune in LJSPEECH data and I can synthesize the sentence but it is not high quality. I will attach my Mel-spectrogram below and please help me to find out how to improve
  2. I want to use your repo to make voice cloning, I am not sure it cannot, so that based on yourTTS I make another loss for speaker similarity. And training from scratch. Is it possible?
hathubkhn commented 2 years ago

kids_are_sitting_on_the_door_and_today_is_very_nice_I_want_to_go_out_Actor_001_neutral

hathubkhn commented 2 years ago

Here is my training Screenshot 2022-09-02 at 9 09 47 AM from scratch when adding speaker loss (SCL-speaker consistance loss) and training with LJSPEECH

keonlee9420 commented 1 year ago

Ah, so sorry for the late response. I thought I replied to your comments.

  1. It might be due to the light weight conv. Replacing it with normal transformer block will resolve the quality issue.
  2. Yes, if the lambda (weight of each loss) is carefully assigned by some experiments.
rsandx commented 1 year ago

@keonlee9420,

About your last point 1 for a potential solution of the quality issue, can you provide an example for replacing the light weight conv. with normal transformer block? Thanks