search

rioyokotalab / caffe2

Caffe2 is a lightweight, modular, and scalable deep learning framework.
https://caffe2.ai
Other
2 stars 0 forks source link

accuracy is fixed to 1 Resnet50 fp16 training Problem #19

Closed Hiroki11x closed 7 years ago

Hiroki11x commented 7 years ago 
<class 'caffe2.python.core.Net'>
{}
<class 'caffe2.python.core.Net'>
{}
<class 'caffe2.python.core.Net'>
{}
<class 'caffe2.python.core.Net'>
{}
INFO:resnet50_trainer:Finished iteration 1/10009 of epoch 0 (25.41 images/sec)
INFO:resnet50_trainer:Training loss: 7.38396549225, accuracy: 0.0
INFO:resnet50_trainer:Finished iteration 2/10009 of epoch 0 (492.02 images/sec)
INFO:resnet50_trainer:Training loss: 190.478805542, accuracy: 0.0
INFO:resnet50_trainer:Finished iteration 3/10009 of epoch 0 (550.15 images/sec)
INFO:resnet50_trainer:Training loss: 723.197265625, accuracy: 0.0
INFO:resnet50_trainer:Finished iteration 4/10009 of epoch 0 (543.48 images/sec)
INFO:resnet50_trainer:Training loss: 704.564941406, accuracy: 0.0
INFO:resnet50_trainer:Finished iteration 5/10009 of epoch 0 (559.24 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 6/10009 of epoch 0 (550.31 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 7/10009 of epoch 0 (545.42 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 8/10009 of epoch 0 (569.45 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 9/10009 of epoch 0 (568.98 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 10/10009 of epoch 0 (543.75 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
INFO:resnet50_trainer:Finished iteration 11/10009 of epoch 0 (550.41 images/sec)
INFO:resnet50_trainer:Training loss: nan, accuracy: 1.0
Hiroki11x commented 7 years ago

it depends on

https://github.com/rioyokotalab/caffe2/issues/23

problem