search

chanyn / HKRM

Hybrid Knowledge Routed Module for Large-scale Object Detection (NerIPS2018)
104 stars 19 forks source link

loss overflow during training..? #18

Open mandal4 opened 5 years ago

mandal4 commented 5 years ago

Hi, i am gonna train HKRM, but i got error about loss. when i train on default hyper-parameters, following error occurs. image It seems to error shows too big for start training. So i changed (init) learning rate to 0.0000000000001 and training starts well. But It is too slow to be optimized. I'd like to ask about the error, what occurs the error and how could i fix it in the code? Thx

guyuchao commented 4 years ago

Hi, i am gonna train HKRM, but i got error about loss. when i train on default hyper-parameters, following error occurs. image It seems to error shows too big for start training. So i changed (init) learning rate to 0.0000000000001 and training starts well. But It is too slow to be optimized. I'd like to ask about the error, what occurs the error and how could i fix it in the code? Thx

What and how many GPU do you use?

mandal4 commented 4 years ago

Thx for reply. I use two TITAN Xp!

Lv-shuai commented 2 years ago

bro，How did you solve this problem？I met the same situation