search

pjreddie / darknet

Convolutional Neural Networks
http://pjreddie.com/darknet/
Other
25.68k stars 21.32k forks source link

Training Issue #524

Open smitshilu opened 6 years ago

smitshilu commented 6 years ago

Hello, I wanted to train YOLO and TINY YOLO on my own dataset. But I am getting either Segmantation Fault or Bus Error 10 or it starts with following step

tiny-yolo-voc-faces layer filters size input output 0 conv 16 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 16 1 max 2 x 2 / 2 416 x 416 x 16 -> 208 x 208 x 16 2 conv 32 3 x 3 / 1 208 x 208 x 16 -> 208 x 208 x 32 3 max 2 x 2 / 2 208 x 208 x 32 -> 104 x 104 x 32 4 conv 64 3 x 3 / 1 104 x 104 x 32 -> 104 x 104 x 64 5 max 2 x 2 / 2 104 x 104 x 64 -> 52 x 52 x 64 6 conv 128 3 x 3 / 1 52 x 52 x 64 -> 52 x 52 x 128 7 max 2 x 2 / 2 52 x 52 x 128 -> 26 x 26 x 128 8 conv 256 3 x 3 / 1 26 x 26 x 128 -> 26 x 26 x 256 9 max 2 x 2 / 2 26 x 26 x 256 -> 13 x 13 x 256 10 conv 512 3 x 3 / 1 13 x 13 x 256 -> 13 x 13 x 512 11 max 2 x 2 / 1 13 x 13 x 512 -> 13 x 13 x 512 12 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 13 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024 14 conv 40 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 40 15 detection mask_scale: Using default '1.000000' Loading weights from tiny-yolo-voc.conv.13...Done! Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005 Loaded: 0.442558 seconds Region Avg IOU: 0.003555, Class: 0.418171, Obj: 0.553781, No Obj: 0.527437, Avg Recall: 0.000000, count: 84 Region Avg IOU: 0.006872, Class: 0.442519, Obj: 0.409464, No Obj: 0.527483, Avg Recall: 0.000000, count: 109 Region Avg IOU: 0.017990, Class: 0.357819, Obj: 0.680002, No Obj: 0.525885, Avg Recall: 0.000000, count: 38 Region Avg IOU: 0.025249, Class: 0.365942, Obj: 0.417638, No Obj: 0.526963, Avg Recall: 0.000000, count: 61 Region Avg IOU: 0.009246, Class: 0.361410, Obj: 0.624200, No Obj: 0.528175, Avg Recall: 0.000000, count: 119 Region Avg IOU: 0.025857, Class: 0.324964, Obj: 0.534475, No Obj: 0.528241, Avg Recall: 0.000000, count: 57 Region Avg IOU: 0.004928, Class: 0.207353, Obj: 0.577038, No Obj: 0.529034, Avg Recall: 0.000000, count: 32 Region Avg IOU: 0.003103, Class: 0.412334, Obj: 0.401760, No Obj: 0.524548, Avg Recall: 0.000000, count: 124 1: 836.332336, 836.332336 avg, 0.001000 rate, 1.170283 seconds, 64 images Loaded: 0.000039 seconds Region Avg IOU: 0.000525, Class: 0.702263, Obj: 0.034110, No Obj: 0.126465, Avg Recall: 0.000000, count: 104 Region Avg IOU: 0.039986, Class: 0.587940, Obj: 0.059002, No Obj: 0.119053, Avg Recall: 0.000000, count: 14 Region Avg IOU: 0.001226, Class: 0.741830, Obj: 0.031014, No Obj: 0.127219, Avg Recall: 0.000000, count: 93 Region Avg IOU: 0.000078, Class: 0.737563, Obj: 0.027721, No Obj: 0.122176, Avg Recall: 0.000000, count: 56 Region Avg IOU: 0.000000, Class: 0.633011, Obj: 0.061815, No Obj: 0.122816, Avg Recall: 0.000000, count: 86 Region Avg IOU: 0.000000, Class: 0.707141, Obj: 0.033334, No Obj: 0.121874, Avg Recall: 0.000000, count: 44 Region Avg IOU: 0.000000, Class: 0.773092, Obj: 0.041615, No Obj: 0.121480, Avg Recall: 0.000000, count: 54 Region Avg IOU: 0.000003, Class: 0.649372, Obj: 0.031467, No Obj: 0.121896, Avg Recall: 0.000000, count: 82

Then it turn to this step

12: nan, nan avg, 0.001000 rate, 1.032995 seconds, 768 images Loaded: 0.000055 seconds Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 35 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 162 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 92 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 85 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 52 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 85 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 79 Region Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000, count: 76

Can you please help me how to train it on my own dataset?

Thank you

alapha23 commented 6 years ago

The reason of segmentation fault might lie in the annotation of your dataset. Perhaps you could have a closer look and make sure the format, range of bounding box are all proper. Also, make sure gpu memory is sufficient could also ease the procedure!