train loss change from normal to NAN

[dtiny]

Provided code : python configuration_10_320_20L_5scales_v2.py Provided data : widerface_train_data_gt_8.pkl At begining, train loss converge normal. iteration times 3400, loss was divergent to nan.
How to solve this problem.

[coderhss]

我也出现了同样的问题

[xinyikb]

same problem +1

[Brain-Lee]

Have you found the inference code

[120276215]

代码有问题：

1. loss写的不对，在难例挖掘那块
2. gray区域在loss里面也没有使用

把1改过来，如不行再降低初始学习率；2可改可不改

[suyue6]

出现同样的问题+1

[suyue6]

代码有问题：

1. loss写的不对，在难例挖掘那块
2. gray区域在loss里面也没有使用

把1改过来，如不行再降低初始学习率；2可改可不改

你好，请教具体怎么改呀，谢谢~~

[Jialeen]

有人解决这个问题了吗

[120276215]

代码有问题：

1. loss写的不对，在难例挖掘那块
2. gray区域在loss里面也没有使用

把1改过来，如不行再降低初始学习率；2可改可不改

你好，请教具体怎么改呀，谢谢~~

https://github.com/becauseofAI/lffd-pytorch/blob/f7da857f7ea939665b81d7bfedb98d02f4147723/ChasingTrainFramework_GeneralOneClassDetection/loss_layer_farm/loss.py#L112

改为： torch.ones_like(pred_score_softmax[:, 1, :, :]).add(1))

[Jialeen]

代码有问题：

1. loss写的不对，在难例挖掘那块
2. gray区域在loss里面也没有使用

把1改过来，如不行再降低初始学习率；2可改可不改

你好，请教具体怎么改呀，谢谢~~

https://github.com/becauseofAI/lffd-pytorch/blob/f7da857f7ea939665b81d7bfedb98d02f4147723/ChasingTrainFramework_GeneralOneClassDetection/loss_layer_farm/loss.py#L112

改为： torch.ones_like(pred_score_softmax[:, 1, :, :]).add(1))

这样修改还是有NAN

[chenjun2hao]

the same problem when training.

[deep-practice]

@becauseofAI Any suggestions?

[Manideep08]

Did anyone find any solution?

[junaiddk]

Anyone found the solution to this problem?

[afterimagex]

无力吐槽，这代码放出来专门坑人的

[CodexForster]

Try reducing the learning rate (variable name = param_learning_rate) to 0.01 in the configuration file. If you are using V2, it should be configuration_10_320_20L_5scales_v2.py. This worked for me to train for 2000000 training loops. EDIT: I see that user 120276215 has already advised the same. So credits to him/her.