search

DL-IT / generative_zoo

54 stars 3 forks source link

Improved WGAN memory leak and training issue #4

Open vishwakftw opened 7 years ago

vishwakftw commented 7 years ago

There seems to be a memory leak in the improved WGAN code. This is due to the BatchNorm2d layers in the Discriminator and Generator of the DCGAN, which is used for the improved WGAN and the torch.autograd.grad function. This is also referenced in this issue: PyTorch Double Backward on BatchNorm2d.

The patch is ready, but we might have to wait until the next release.

There seems to be problem with training without the BatchNorm2d layers. This needs to be tracked and fixed.

vishwakftw commented 7 years ago

The problem now is that the training is extremely unstable. The D-loss and G-loss are decreasing constantly and don't seem to flatten out/saturate.