search

Sense-X / Co-DETR

[ICCV 2023] DETRs with Collaborative Hybrid Assignments Training
MIT License
968 stars 105 forks source link

Box AP of new ViT-L model doesn't match #79

Closed DianCh closed 10 months ago

DianCh commented 10 months ago

Hi! I evaluated the newly release ViT-L model on LVIS with the following command:

bash tools/dist_test.sh projects/configs/co_dino/co_dino_5scale_vit_large_lvis.py data/co_dino_5scale_vit_large_lvis.pth 2 --eval bbox

and the performance doesn't match what's released in the table:

Evaluating bbox...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.644
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.795
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.694
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.559
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.741
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.786
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.600
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.656
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.649
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.782
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.674
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.870
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.912
OrderedDict([('bbox_AP', 0.644), ('bbox_AP50', 0.795), ('bbox_AP75', 0.694), ('bbox_APs', 0.559), ('bbox_APm', 0.741), ('bbox_APl', 0.786), ('bbox_APr', 0.6), ('bbox_APc', 0.656), ('bbox_APf', 0.649), ('bbox_mAP_copypaste', 'AP:0.644 AP50:0.795 AP75:0.694 APs:0.559 APm:0.741 APl:0.786 APr:0.600 APc:0.656 APf:0.649')])

What did I miss? Thanks

TempleX98 commented 10 months ago

Sorry, I found a bug in the config. The attention mask should be disabled:

model = dict(
    type='CoDETR',
    with_attn_mask=False,
    backbone=dict(
DianCh commented 10 months ago

Thank you! The fix worked for me. Now I get

Evaluating bbox...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.680
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.812
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.723
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.589
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.774
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.822
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.645
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.689
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.685
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.806
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.699
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.892
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.936
OrderedDict([('bbox_AP', 0.68), ('bbox_AP50', 0.812), ('bbox_AP75', 0.723), ('bbox_APs', 0.589), ('bbox_APm', 0.774), ('bbox_APl', 0.822), ('bbox_APr', 0.645), ('bbox_APc', 0.689), ('bbox_APf', 0.685), ('bbox_mAP_copypaste', 'AP:0.680 AP50:0.812 AP75:0.723 APs:0.589 APm:0.774 APl:0.822 APr:0.645 APc:0.689 APf:0.685')])