训练了4000iters，验证完之后卡着了，请问这是怎么回事呢？

[mary-0830]

2022-07-07 14:05:26,893 - mmdet.ssod - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.135 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.269 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.120 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.071 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.155 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.164 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.119 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.281 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.331

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 155.5 task/s, elapsed: 32s, ETA: 0s2022-07-07 14:06:02,265 - mmdet.ssod - INFO - Evaluating bbox... Loading and preparing results... DONE (t=0.11s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=14.59s). Accumulating evaluation results... DONE (t=3.20s). 2022-07-07 14:06:20,482 - mmdet.ssod - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.088 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.193 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.068 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.045 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.105 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.112 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.063 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.186 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.216

2022-07-07 14:06:20,850 - mmdet.ssod - INFO - Exp name: soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k.py 2022-07-07 14:06:20,855 - mmdet.ssod - INFO - Iter(val) [4000] teacher.bbox_mAP: 0.1350, teacher.bbox_mAP_50: 0.2690, teacher.bbox_mAP_75: 0.1200, teacher.bbox_mAP_s: 0.0710, teacher.bbox_mAP_m: 0.1550, teacher.bbox_mAP_l: 0.1640, teacher.bbox_mAP_copypaste: 0.135 0.269 0.120 0.071 0.155 0.164, student.bbox_mAP: 0.0880, student.bbox_mAP_50: 0.1930, student.bbox_mAP_75: 0.0680, student.bbox_mAP_s: 0.0450, student.bbox_mAP_m: 0.1050, student.bbox_mAP_l: 0.1120, student.bbox_mAP_copypaste: 0.088 0.193 0.068 0.045 0.105 0.112

请问跑完了4000iters以后，程序就卡着不动了，这是咋回事呢？

[herfootsosmell]

您好，我也遇到了同样的问题，请问您解决了吗

[Whale-ice]

请问，你们解决了吗，我也遇到这个问题了 AssertionError: Default process group is not initialized

[Re-dot-art]

2022-07-07 14:05:26,893 - mmdet.ssod - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.135 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.269 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.120 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.071 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.155 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.164 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.262 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.119 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.281 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.331

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 155.5 task/s, elapsed: 32s, ETA: 0s2022-07-07 14:06:02,265 - mmdet.ssod - INFO - Evaluating bbox... Loading and preparing results... DONE (t=0.11s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=14.59s). Accumulating evaluation results... DONE (t=3.20s). 2022-07-07 14:06:20,482 - mmdet.ssod - INFO - Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.088 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.193 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.068 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.045 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.105 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.112 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.169 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.063 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.186 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.216

2022-07-07 14:06:20,850 - mmdet.ssod - INFO - Exp name: soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k.py 2022-07-07 14:06:20,855 - mmdet.ssod - INFO - Iter(val) [4000] teacher.bbox_mAP: 0.1350, teacher.bbox_mAP_50: 0.2690, teacher.bbox_mAP_75: 0.1200, teacher.bbox_mAP_s: 0.0710, teacher.bbox_mAP_m: 0.1550, teacher.bbox_mAP_l: 0.1640, teacher.bbox_mAP_copypaste: 0.135 0.269 0.120 0.071 0.155 0.164, student.bbox_mAP: 0.0880, student.bbox_mAP_50: 0.1930, student.bbox_mAP_75: 0.0680, student.bbox_mAP_s: 0.0450, student.bbox_mAP_m: 0.1050, student.bbox_mAP_l: 0.1120, student.bbox_mAP_copypaste: 0.088 0.193 0.068 0.045 0.105 0.112

请问跑完了4000iters以后，程序就卡着不动了，这是咋回事呢？

How do you configure the environment? I have been trying for a long time but have not been successful. Can you share your method?

[Re-dot-art]

您好，我也遇到了同样的问题，请问您解决了吗

请问您解决了这个问题吗