search

JasonSWFu / MetricGAN

MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (ICML 2019, with Travel awards)
131 stars 34 forks source link

~loss: Nan #10

Open LexiYIN opened 3 years ago

LexiYIN commented 3 years ago

Hi，Jason：

When I try to train the MetricGAN(table2).py, I met two problems: (1)At every epoch both on G and D, its loss is nan , but only show a warning :

/student/home/yll/anaconda3/envs/MetricGAN/lib/python3.6/site-packages/keras/engine/training.py:973: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?
  'Discrepancy between trainable weights and collected trainable'
Epoch 1/1
 - 1498s - loss: nan

(2) At the fisrt epoch of traning Discriminator, it taken 593M on GPU's memory but its ultilization is 0%.

Data info: I use the same dataset from SEGAN already downsampled to 16k by the way, keras-gpu=2.1.2, tensorflow-gpu=1.10, librosa=0.5.1, python=3.6

Do you know why this happen（loss nan）？ Thank you very much.

JasonSWFu commented 3 years ago

HI,

I never met this nan problem before. I'm not sure whether it is due to a different python version (mine is Python 2.7) or PESQ code. Maybe you can also try to use TargetMetric='stoi' to test the code.

LexiYIN commented 3 years ago

Hi，Could you tell me the version of your tensorflow ,please.

JasonSWFu commented 3 years ago

Hi, it is 1.4.0