Training starts but no updates are dumping to checkpoints?

chrisdonahue / wavegan

WaveGAN: Learn to synthesize raw audio with generative adversarial networks

MIT License

1.33k stars 282 forks source link

Training starts but no updates are dumping to checkpoints? #112

Open JordanDavis-1 opened 2 years ago

JordanDavis-1 commented 2 years ago

Not sure if this is a wavegan related problem or my own install of tensorflow / python. The training will start but only dumps "model.ckpt-0" etc.. and never updates.

Any ideas? Thanks!

markhanslip commented 2 years ago

Hey Jordan,

I've just revisited this codebase and I'm having a similar problem. It was working fine for me middle of last year but now it progresses incredibly slowly - checkpoints are being dumped but at a dramatically lower rate than previously. What's confusing is that the model is clearly inhabiting the GPU but is not being trained at GPU-like speeds. The same behaviour is happening locally and in Colab.

I'll update if I manage to figure out a workaround or cause.

Cheers