search

chrisdonahue / wavegan

WaveGAN: Learn to synthesize raw audio with generative adversarial networks
MIT License
1.33k stars 281 forks source link

v2 code could not train #34

Closed yumath closed 5 years ago

yumath commented 5 years ago

I only can run the v1 code, v2 code would be stuck when training Discriminator.

chrisdonahue commented 5 years ago

You need to provide more information

yumath commented 5 years ago

In the same environment, tensorflow-1.12 and Nvidia GTX 1080 Ti. with the same dataset, sc09 v1:tf_records(path:data/sc09/all tf_records files) vs v2:raw WAV files(path:data/sc09/train/all raw WAV files) v1 is trainable When I started training v2, it runs a long time and nothing output. So I used a print function during training v2, print every time before training Discriminator. However, the print function would only be executed one time. And the program still be running, I waited for 10 hours and the program did not print again.

ExtraLime commented 5 years ago

Hi yumath did you find a fix for this issue? I seem to be having the same problem.

yumath commented 5 years ago

Hi yumath did you find a fix for this issue? I seem to be having the same problem.

I have not solved this problem yet. And I think it may be a problem with data loading because the two versions of the model code are basically the same.  

ExtraLime commented 5 years ago

Ok ill look into that. let you know if i can find anything.

yumath commented 5 years ago

Thanks a lot!

chrisdonahue commented 5 years ago

Even during successful training, nothing will print out of the console. Please use tensorboard to monitor the log directory and let me know if anything appears there.

yumath commented 5 years ago

Even during successful training, nothing will print out of the console. Please use tensorboard to monitor the log directory and let me know if anything appears there.

When training v2, I used Tensorboard, but it only displays "Graph" and "Project", cannot see "Scalar" or "Histogram" and "Audio".

ExtraLime commented 5 years ago

yumath, this was my error. In order to train the SC09 data, I reread the instructions, should add '--data_first_slice' for this dataset.

yumath commented 5 years ago

yumath, this was my error. In order to train the SC09 data, I reread the instructions, should add '--data_first_slice' for this dataset.

thank you! I tried it and really works.

amjltc295 commented 5 years ago

I had the same problem and fixed it by adding --data_first_slice. Without --data_first_slice the tensorboard just shows "Graph" and "Project" without any error messages. I think it would be nice to raise an error if the model is not reading data correctly.

chrisdonahue commented 5 years ago

I agree that it would be nice to error out in this case. I'm not entirely sure how to do that unfortunately. The file I/O is handled within the Tensorflow graph so it's nontrivial. Feel free to make a PR if you can figure it out :)