search

barisozmen / deepaugment

Discover augmentation strategies tailored for your dataset
MIT License
244 stars 41 forks source link

Loss turns into 'nan' #18

Open barisozmen opened 5 years ago

barisozmen commented 5 years ago

I'm experimenting with wrn-16-8 (WideResNet) at this repo. During training, loss suddenly turned into nan. I guess it's a numerical calculation problem.

screen shot 2019-02-03 at 12 29 36 am
barisozmen commented 5 years ago

Setting clipnorm to 1. was suggested at https://stackoverflow.com/questions/37232782/nan-loss-when-training-regression-network

It didn't worked..

lichuanx commented 5 years ago

Might some issues from data.