Open Smoothing97 opened 5 years ago
@ringringyi 您好！很感谢您的代码，训练过程中出现了一些问题： 1.按照说明加载了yolo-dota.cfg，只有单卡GPU,NVIDIA GT TITAN BLACK(算力3.5），训练1024大小的DOTA train数据集，因此设置： batch=1 subdivisions=1 网络cfg中学习率是 learning_rate=0.00005 max_batches = 2000000 policy=constant 训练时已经加载了darknet19_448.conv.23。 训练约12h，共训练了109000张图然后停止了训练，因为得到的训练结果如下： Region Avg IOU: 0.618628, Class: 0.671129, Obj: 0.088273, No Obj: 0.003471, Avg Recall: 0.666667, count: 3 Region Avg IOU: 0.350919, Class: 0.315243, Obj: 0.000529, No Obj: 0.002533, Avg Recall: 0.500000, count: 2 Region Avg IOU: 0.507692, Class: 0.390687, Obj: 0.003199, No Obj: 0.001610, Avg Recall: 0.600000, count: 5 Region Avg IOU: 0.586214, Class: 0.206978, Obj: 0.030189, No Obj: 0.002246, Avg Recall: 1.000000, count: 3 Region Avg IOU: 0.744804, Class: 0.428495, Obj: 0.018022, No Obj: 0.003115, Avg Recall: 1.000000, count: 2 Region Avg IOU: 0.219129, Class: 0.057150, Obj: 0.000013, No Obj: 0.002348, Avg Recall: 0.000000, count: 2
109280: 169.920197, 45.703640 avg, 0.000050 rate, 0.414891 seconds, 109280 images 109281: 26.782028, 43.811478 avg, 0.000050 rate, 0.408204 seconds, 109281 images 109282: 20.342304, 41.464561 avg, 0.000050 rate, 0.419111 seconds, 109282 images 109283: 24.117226, 39.729828 avg, 0.000050 rate, 0.408410 seconds, 109283 images 109284: 26.526783, 38.409523 avg, 0.000050 rate, 0.413556 seconds, 109284 images 109285: 15.803896, 36.148960 avg, 0.000050 rate, 0.419731 seconds, 109285 images 109286: 50.919559, 37.626019 avg, 0.000050 rate, 0.411063 seconds, 109286 images 109287: 0.020407, 33.865456 avg, 0.000050 rate, 0.420056 seconds, 109287 images 109288: 29.913115, 33.470222 avg, 0.000050 rate, 0.413893 seconds, 109288 images 109289: 28.218138, 32.945015 avg, 0.000050 rate, 0.425859 seconds, 109289 images 109290: 0.749020, 29.725416 avg, 0.000050 rate, 0.407215 seconds, 109290 images 109291: 0.018226, 26.754698 avg, 0.000050 rate, 0.415694 seconds, 109291 images 109292: 12.341995, 25.313427 avg, 0.000050 rate, 0.411180 seconds, 109292 images
我将输出结果分开了，第一部分去掉了报nan的行（大约有30%报nan)，cls和Obj在训练了10w张图片后仍然结果不高；第二部分的loss值和avg loss值在30以上。 2.停止训练后用10w次训练得到的weights对一张416大小的图像做了测试，效果不佳。下图是thresh=0.1的输出。 thresh>0.1时没有输出。 想请教这种状况是正常的吗，怎么可以调整？
@ringringyi 您好！很感谢您的代码，训练过程中出现了一些问题： 1.按照说明加载了yolo-dota.cfg，只有单卡GPU,NVIDIA GT TITAN BLACK(算力3.5），训练1024大小的DOTA train数据集，因此设置： batch=1 subdivisions=1 网络cfg中学习率是 learning_rate=0.00005 max_batches = 2000000 policy=constant 训练时已经加载了darknet19_448.conv.23。 训练约12h，共训练了109000张图然后停止了训练，因为得到的训练结果如下： Region Avg IOU: 0.618628, Class: 0.671129, Obj: 0.088273, No Obj: 0.003471, Avg Recall: 0.666667, count: 3 Region Avg IOU: 0.350919, Class: 0.315243, Obj: 0.000529, No Obj: 0.002533, Avg Recall: 0.500000, count: 2 Region Avg IOU: 0.507692, Class: 0.390687, Obj: 0.003199, No Obj: 0.001610, Avg Recall: 0.600000, count: 5 Region Avg IOU: 0.586214, Class: 0.206978, Obj: 0.030189, No Obj: 0.002246, Avg Recall: 1.000000, count: 3 Region Avg IOU: 0.744804, Class: 0.428495, Obj: 0.018022, No Obj: 0.003115, Avg Recall: 1.000000, count: 2 Region Avg IOU: 0.219129, Class: 0.057150, Obj: 0.000013, No Obj: 0.002348, Avg Recall: 0.000000, count: 2
109280: 169.920197, 45.703640 avg, 0.000050 rate, 0.414891 seconds, 109280 images 109281: 26.782028, 43.811478 avg, 0.000050 rate, 0.408204 seconds, 109281 images 109282: 20.342304, 41.464561 avg, 0.000050 rate, 0.419111 seconds, 109282 images 109283: 24.117226, 39.729828 avg, 0.000050 rate, 0.408410 seconds, 109283 images 109284: 26.526783, 38.409523 avg, 0.000050 rate, 0.413556 seconds, 109284 images 109285: 15.803896, 36.148960 avg, 0.000050 rate, 0.419731 seconds, 109285 images 109286: 50.919559, 37.626019 avg, 0.000050 rate, 0.411063 seconds, 109286 images 109287: 0.020407, 33.865456 avg, 0.000050 rate, 0.420056 seconds, 109287 images 109288: 29.913115, 33.470222 avg, 0.000050 rate, 0.413893 seconds, 109288 images 109289: 28.218138, 32.945015 avg, 0.000050 rate, 0.425859 seconds, 109289 images 109290: 0.749020, 29.725416 avg, 0.000050 rate, 0.407215 seconds, 109290 images 109291: 0.018226, 26.754698 avg, 0.000050 rate, 0.415694 seconds, 109291 images 109292: 12.341995, 25.313427 avg, 0.000050 rate, 0.411180 seconds, 109292 images
我将输出结果分开了，第一部分去掉了报nan的行（大约有30%报nan)，cls和Obj在训练了10w张图片后仍然结果不高；第二部分的loss值和avg loss值在30以上。 2.停止训练后用10w次训练得到的weights对一张416大小的图像做了测试，效果不佳。下图是thresh=0.1的输出。
thresh>0.1时没有输出。
想请教这种状况是正常的吗，怎么可以调整？