search

JiahuiYu / wdsr_ntire2018

Code of our winning entry to NTIRE super-resolution challenge, CVPR 2018
http://www.vision.ee.ethz.ch/ntire18/
598 stars 123 forks source link

loss suddenly turned into nan #36

Closed real-zhangzhe closed 5 years ago

real-zhangzhe commented 5 years ago

Hi, when I train with lr=1e-4, batch_size=16, out_patch_size=192, my loss was normal, after about 15,000 steps (1 epoch = 30,000 steps), the loss suddenly turned into nan, my data was clean, and I used L1Loss. How should this problem be solved? thanks

JiahuiYu commented 5 years ago

@Z-zhe In that case for your dataset, I would suggest lower learning rate, maybe by half. Another option is to use warm starting learning rates.

https://arxiv.org/abs/1706.02677