VGG gradients explode significantly after epoch one

Nikronic / CoarseNet

Modified version of U-net based on "Convolutional Networks for Biomedical Image Segmentation (Ronneberger et al., 2015)" paper.

https://arxiv.org/abs/1505.04597

MIT License

25 stars 2 forks source link

VGG gradients explode significantly after epoch one #8

Closed Nikronic closed 5 years ago

Nikronic commented 5 years ago

After passing epoch one, suddenly gradients explode about 10^30 times. I addressed a good implemenation EnhanceNet using Tensorflow in loss function at MSE loss definition line.

Nikronic commented 5 years ago

Please see commit https://github.com/Nikronic/CoarseNet/commit/3382bd3760f0be9078eb0827dc2141af06f547f5

Nikronic commented 5 years ago

Problem solved. The problem caused when I tried to run the model on my local environment and using the CPU. Everything sounds fine on the GPU based on the test on Google colab. See commit https://github.com/Nikronic/CoarseNet/commit/b62c4644cbb5584518f1829f30983af0e270e27a .