Custom training: All nan with 0 counts

[omar391]

I have marked all the images with Yolo_mark tools and create a weight with yolov3.cfg. However, when I tried to use the tiny config ( I followed this: https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects) the training goes wrong. ALL NAN and 0 counts

Config: yolov3-tiny_obj.cfg.zip data+names.zip

• Total classes: 6
• Its a GPU training Sample data inside text:

1 0.565234 0.547222 0.108594 0.094444 0 0.680078 0.549306 0.097656 0.106944 0 0.261328 0.509722 0.052344 0.080556 0 0.286719 0.470833 0.048437 0.075000 0 0.257422 0.440972 0.049219 0.068056

What should I do? I have tried to check similar issues with no luck.

[omar391]

@AlexeyAB Could you please help with my issue..

[BlcaKHat]

for me also, after 900 iteration, the avg loss become 0.000 and average IOU is nan. do you have a local machine for training

[omar391]

I am using a local machine for training. I have avg loss =0, all NAN, all count =0. So it seems there must a problem somewhere.

[mohhao]

I have the same problem,have you solved it?

[omar391]

No, I am still trying. I am currently checking if any of my yolo mark annotations is wrong and then recalculate anchors,

[dexception]

Use yolo_mark tool and check your dataset.

[AlexeyAB]

• What params do you use in the Makefile?

• And check files bad.list and bad_label.list

• At what iteration does the loss become zero?

• What command did you use for training?

I just trained model with your cfg-file and my dataset, and I got avg loss 0.48 after 3000 iterations:

[clydebailey]

What is bad.list and bad_label.list?