search

eg4000 / SKU110K_CVPR19

767 stars 181 forks source link

Why the loss can't reduce when i train the IoU layer? #8

Closed MengNan-Li closed 5 years ago

MengNan-Li commented 5 years ago

When i train the IOU layer, the loss maintained at around 0.4. 1438/10000 [===>..........................] - ETA: 3:57:13 - loss: 0.4222 1439/10000 [===>..........................] - ETA: 3:57:10 - loss: 0.4222 1440/10000 [===>..........................] - ETA: 3:57:07 - loss: 0.4222 1441/10000 [===>..........................] - ETA: 3:57:08 - loss: 0.4222 1442/10000 [===>..........................] - ETA: 3:57:00 - loss: 0.4222 1443/10000 [===>..........................] - ETA: 3:56:58 - loss: 0.4222 1444/10000 [===>..........................] - ETA: 3:56:56 - loss: 0.4223 1445/10000 [===>..........................] - ETA: 3:56:54 - loss: 0.4223 1446/10000 [===>..........................] - ETA: 3:56:51 - loss: 0.4223 1447/10000 [===>..........................] - ETA: 3:56:51 - loss: 0.4223 1448/10000 [===>..........................] - ETA: 3:56:50 - loss: 0.4222 1449/10000 [===>..........................] - ETA: 3:56:52 - loss: 0.4223 1450/10000 [===>..........................] - ETA: 3:56:52 - loss: 0.4223 1451/10000 [===>..........................] - ETA: 3:56:50 - loss: 0.4223 1452/10000 [===>..........................] - ETA: 3:56:49 - loss: 0.4223 1453/10000 [===>..........................] - ETA: 3:56:45 - loss: 0.4223 1454/10000 [===>..........................] - ETA: 3:56:44 - loss: 0.4223 1455/10000 [===>..........................] - ETA: 3:56:43 - loss: 0.4223 1456/10000 [===>..........................] - ETA: 3:56:44 - loss: 0.4222 1457/10000 [===>..........................] - ETA: 3:56:41 - loss: 0.4222 1458/10000 [===>..........................] - ETA: 3:56:38 - loss: 0.4223 1459/10000 [===>..........................] - ETA: 3:56:38 - loss: 0.4223 1460/10000 [===>..........................] - ETA: 3:56:35 - loss: 0.4223 1461/10000 [===>..........................] - ETA: 3:56:35 - loss: 0.4223 1462/10000 [===>..........................] - ETA: 3:56:34 - loss: 0.4224 1463/10000 [===>..........................] - ETA: 3:56:32 - loss: 0.4223 1464/10000 [===>..........................] - ETA: 3:56:30 - loss: 0.4224 1465/10000 [===>..........................] - ETA: 3:56:27 - loss: 0.4224 1466/10000 [===>..........................] - ETA: 3:56:25 - loss: 0.4224 1467/10000 [===>..........................] - ETA: 3:56:23 - loss: 0.4224 1468/10000 [===>..........................] - ETA: 3:56:20 - loss: 0.4224 1469/10000 [===>..........................] - ETA: 3:56:18 - loss: 0.4224 image

I have already trained 23 epochs. image

HoracceFeng commented 5 years ago

Hi @MengNan-Li , have you got a useful model? I also train a model by this code, very slow. In the train-iou session, I train just 2 epoch and the loss is stable at 0.426. Similar circumstance as yours. I then do the prediction based on this model but got nothing. @eg4000 I think we need help on training method, etc ...

eg4000 commented 5 years ago

Hi,

In our experiments, most of the information for IoU loss was indeed learned during the first epochs.

We managed to decrease the loss in the past by changing the threshold of positive examples, the weight balance between weak positive and hard positive examples, and the learning rate strategies. However, in our recent tests we found that the overall performance was not significantly affected using models which converged to a loss values around 0.4. In either case, there is certainly room for further research.

Regards, Eran.

HoracceFeng commented 5 years ago

Hi Eran, Thanks for the reply, looking forward to your updates.

Sent from my iPhone

On Jun 14, 2019, at 03:42, eg4000 notifications@github.com<mailto:notifications@github.com> wrote:

Hi,

In our experiments, most of the information for IoU loss was indeed learned during the first epochs.

We managed to decrease the loss in the past by changing the threshold of positive examples, the weight balance between weak positive and hard positive examples, and the learning rate strategies. However, in our recent tests we found that the overall performance was not significantly affected using models which converged to a loss values around 0.4. In either case, there is certainly room for further research.

Regards, Eran.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feg4000%2FSKU110K_CVPR19%2Fissues%2F8%3Femail_source%3Dnotifications%26email_token%3DAE6372IAC5LQQBJABLG7QJLP2KPKRA5CNFSM4HW2BABKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXU2I4A%23issuecomment-501851248&data=02%7C01%7C%7C3da48f40b9054a45e3b108d6f0374742%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636960517533701814&sdata=C7wI22rHxzBrIqMk15fE2uBy8rBTWT%2BpXLuqc9FMzbw%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAE6372NHFYGAPFJXJP7FVUDP2KPKRANCNFSM4HW2BABA&data=02%7C01%7C%7C3da48f40b9054a45e3b108d6f0374742%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636960517533711825&sdata=5cM14U4jgi%2F%2FLRKX3VoyOeWomlJeNtyWNRBekbpbe7Q%3D&reserved=0.