search

SysCV / shift-detection-tta

This repository implements continuous test-time adaptation algorithms for object detection on the SHIFT dataset.
MIT License
18 stars 2 forks source link

Questions about the results of the experiment #5

Closed zheng-chen777 closed 12 months ago

zheng-chen777 commented 12 months ago

When I run the validation script: bash scripts/continuous/no_adap_yolox/val_yolox_shift_from_clear_daytime.sh

07/09 11:08:22 - mmengine - INFO - load model from: https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_x_8x8_300e_coco/yolox_x_8x8_300e_coco_20211126_140254-1ef88d67.pth
07/09 11:08:22 - mmengine - INFO - Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_x_8x8_300e_coco/yolox_x_8x8_300e_coco_20211126_140254-1ef88d67.pth
07/09 11:08:22 - mmengine - WARNING - The model and loaded state dict do not match exactly

size mismatch for bbox_head.multi_level_conv_cls.0.weight: copying a param with shape torch.Size([80, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([6, 320, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([6]).
size mismatch for bbox_head.multi_level_conv_cls.1.weight: copying a param with shape torch.Size([80, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([6, 320, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([6]).
size mismatch for bbox_head.multi_level_conv_cls.2.weight: copying a param with shape torch.Size([80, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([6, 320, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([6]).

This may be because the MMDET version we are currently using is version 3.0.0rc5, but the model version of Pretrain is MMDET 2.0. I wondered if that would have an impact on the results, and I got the results below, and I wanted to know if they were consistent with your results

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.357
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.529
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.409
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.248
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.692
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.722
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.337
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.740
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.761
07/09 11:04:37 - mmengine - INFO - 
+------------+-------+------------+-------+----------+-------+
| category   | AP    | category   | AP    | category | AP    |
+------------+-------+------------+-------+----------+-------+
| pedestrian | 0.396 | car        | 0.466 | truck    | 0.462 |
| bus        | 0.352 | motorcycle | 0.41  | bicycle  | 0.059 |
+------------+-------+------------+-------+----------+-------+
07/09 11:04:37 - mmengine - INFO - bbox_mAP_copypaste: 0.357 0.529 0.409 0.248 0.692 0.722
07/09 11:04:37 - mmengine - INFO - Epoch(test) [2400/2400]    coco/pedestrian_precision: 0.3960  coco/car_precision: 0.4660  coco/truck_precision: 0.4620  coco/bus_precision: 0.3520  coco/motorcycle_precision: 0.4100  coco/bicycle_precision: 0.0590  coco/bbox_mAP: 0.3570  coco/bbox_mAP_50: 0.5290  coco/bbox_mAP_75: 0.4090  coco/bbox_mAP_s: 0.2480  coco/bbox_mAP_m: 0.6920  coco/bbox_mAP_l: 0.7220  data_time: 0.0021  time: 0.0573
mattiasegu commented 12 months ago

Dear @zheng-chen777, the results are indeed correct.

as far as I understand, there is no incompatibility issue with your packages. the WARNING message you mention just pops up because we are first loading the YOLOX checkpoint pre-trained on COCO, but SHIFT has a different number of classes. After that warning message, the repository also loads our pretrained checkpoint on SHIFT and your results indeed match ours.

Please let me know if you have other questions!