search

bill9800 / speech_separation

Include some core functions and model to handle speech separation
MIT License
155 stars 61 forks source link

Question about the training loss #7

Open xiaoyiming opened 5 years ago

xiaoyiming commented 5 years ago

@bill9800 Hi, It's really a amazing work. Thanks for sharing the code. However, I have some problems about the training loss. I trained 9 epoch ( datasets about 30000 videos,batchsize = 2. ). I noticed that my original training loss was about 0.45. After 9 epoch, my training loss is about 0.18 and it can not decrease. It is normal? what is the situation about your training loss? I am looking for your replay!

bill9800 commented 5 years ago

Hi, I try some approaches to avoid that situation. However, I did not have enough resource and time to train more than 20 epochs, not sure what will happen after then.
Here are some possible solutions to the problem.

  1. You can try to modify the learning rate scheduler part.
  2. I also have an idea that maybe you can add some auxiliary output layer before the last layer (like googLenet).
  3. Add dropout layer
  4. modify loss function (or change the gamma, increase it will increase the influence of the difference between two output) I think what is also important is to check the result by listening to it. Less loss does not guarantee a good result based on my experience.
xiaoyiming commented 5 years ago

@bill9800 Thanks for replay. And I check the result by listening to it. I create much losses, and dot not separate the two voices. I will try your advice. Do you separate the two voices after 20 epochs? what is your train and vail loss?

bill9800 commented 5 years ago

Hi, My result is not always as good as Goolge's paper even if the loss is low. Some samples did well but some are not. Sorry that I cannot provide the experiment data for you.

I would suggest not to compare the loss between different gamma. There are many ways to check the performance, one simple way is to compare the SNR of the result.

xiaoyiming commented 5 years ago

Thanks for replay !! I understand you that cannot the experiment data ? However could you please share me your batch-size and how many data used in your training

83344rushikesh commented 4 years ago

@xiaoyiming Do you have your working audio-visual model?

vuthede commented 4 years ago

@xiaoyiming Hi, How is it going with your work in here? have you made the model better, I have started research in this work and started by trying to reproduce the result, but it seems the people have been suffering to reproduce this work. Thanks