Bohao-Lee / CME

65 stars 15 forks source link

occur the NAN when training the Net using the pascal voc #30

Open SamXiaosheng opened 2 years ago

SamXiaosheng commented 2 years ago

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000 coord_mask: tensor(27.0292, device='cuda:0') 80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan coord_mask: tensor(23.9451, device='cuda:0') 96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

shizhongwu commented 1 year ago

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000 coord_mask: tensor(27.0292, device='cuda:0') 80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan coord_mask: tensor(23.9451, device='cuda:0') 96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

我也遇到了同样的问题,我想问一下这个问题你解决了吗?

The-Hulk-007 commented 6 months ago

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000 coord_mask: tensor(27.0292, device='cuda:0') 80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan coord_mask: tensor(23.9451, device='cuda:0') 96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

我也遇到了同样的问题,我想问一下这个问题你解决了吗?

[降低学习率试试,