search

Bohao-Lee / CME

64 stars 15 forks source link

all are nan #11

Open pujianjian opened 3 years ago

pujianjian commented 3 years ago

Hello, I follow your process, but during the training, after a while, all are nan . I'm very upset 1629353991(1)

Bohao-Lee commented 3 years ago

Hello, I follow your process, but during the training, after a while, all are nan . I'm very upset 1629353991(1)

I have met this bug before. Maybe different GPU may causes this bug. You can modify cfg/darknet_dynamic.cfg

steps=-1,500,40000,60000
scales=0.1,10,.1,.1

to

steps=-1,500,40000,60000
scales=0.01,100,.1,.1

can solve this bug.

The-Hulk-007 commented 6 months ago

At the beginning, the learning rate == 0.001, and at the beginning of training, proposals == 50000. After a while, proposals == 0, and then "nan" appeared. I am so sad

Later, I modified batchsize="32", learning rate =0.0003, and just started training proposals==50000. After a little longer than last time, proposals==0 appeared, and "nan" did not appear.

一开始学习率==0.001，刚开始训练proposals==50000，过一会proposals==0，然后出现 “nan”。我好伤心

后面，我修改了batchsize="32"，learning rate =0.0003，刚开始训练proposals==50000，过了比上次更久一点，出现proposals==0，没有出现“nan” start as time increases

sad help，being Very impatient