Closed GulpFire closed 1 year ago
`2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016
2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001 `
Me too, How did you solve it?
2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001
Me too, How did you solve it?
我没能复现,但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet
2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001
Me too, How did you solve it?
我没能复现,但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet mfd单卡训练降低学习率试试
8个gpu完全一样的配置文件,loss训练几轮之后就nan了,[FormulaNet]的数据集太难搞了
2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016 2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001
Me too, How did you solve it?
我没能复现,但你或许可以试试这个方案https://github.com/felix-schmitt/FormulaNet mfd单卡训练降低学习率试试
我试过调整学习率,放大或缩小一个数量级,但并没有什么效果
@GulpFire @ZeroCodePro Hi, the bug was fixed, please pull the latest config file and take it a try.
`2022-11-20 01:35:49,834 - mmdet - INFO - Epoch [24][2000/2174] lr: 1.000e-05, eta: 0:05:54, time: 2.101, data_time: 0.014, memory: 15795, loss_cls: 0.2582, loss_bbox: 1.2449, loss_dfl: 0.4970, loss: 2.0000 2022-11-20 01:37:34,752 - mmdet - INFO - Epoch [24][2050/2174] lr: 1.000e-05, eta: 0:04:12, time: 2.098, data_time: 0.014, memory: 15795, loss_cls: 0.2564, loss_bbox: 1.2549, loss_dfl: 0.4957, loss: 2.0070 2022-11-20 01:39:19,524 - mmdet - INFO - Epoch [24][2100/2174] lr: 1.000e-05, eta: 0:02:30, time: 2.095, data_time: 0.014, memory: 15795, loss_cls: 0.2651, loss_bbox: 1.2299, loss_dfl: 0.4924, loss: 1.9874 2022-11-20 01:41:04,317 - mmdet - INFO - Epoch [24][2150/2174] lr: 1.000e-05, eta: 0:00:48, time: 2.096, data_time: 0.013, memory: 15795, loss_cls: 0.2600, loss_bbox: 1.2279, loss_dfl: 0.4954, loss: 1.9832 2022-11-20 01:41:54,811 - mmdet - INFO - Saving checkpoint at 24 epochs 2022-11-20 01:44:58,555 - mmdet - INFO - Evaluating bbox... 2022-11-20 01:45:01,473 - mmdet - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.014 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.016
2022-11-20 01:45:01,502 - mmdet - INFO - Exp name: gfl_s50_fpn_2x_coco.py 2022-11-20 01:45:01,502 - mmdet - INFO - Epoch(val) [24][380] bbox_mAP: 0.0000, bbox_mAP_50: 0.0000, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.001 `