How to decode GQA evaluation results

[yijingru]

Hi I followed this instruction by loading the gqa_resnet101_checkpoint.pth, my result is like the following, do you know how to get the result reported in the paper?

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
{'test_gqa_loss': 364.3319328805411, 'test_gqa_loss_answer_attr': 0.5937237836435415, 'test_gqa_loss_answer_cat': 0.40440608162500535, 'test_gqa_loss_answer_global': 0.03414623715799758, 'test_gqa_loss_answer_obj': 0.03846898033370753, 'test_gqa_loss_answer_rel': 1.462530900234227, 'test_gqa_loss_answer_type': 0.000580944533996148, 'test_gqa_loss_bbox': 0.0, 'test_gqa_loss_bbox_0': 0.0, 'test_gqa_loss_bbox_1': 0.0, 'test_gqa_loss_bbox_2': 0.0, 'test_gqa_loss_bbox_3': 0.0, 'test_gqa_loss_bbox_4': 0.0, 'test_gqa_loss_ce': 2.1158246287386064, 'test_gqa_loss_ce_0': 157.7743893561189, 'test_gqa_loss_ce_1': 96.71868126752456, 'test_gqa_loss_ce_2': 14.313661651126152, 'test_gqa_loss_ce_3': 20.294675855682083, 'test_gqa_loss_ce_4': 70.58084329565682, 'test_gqa_loss_giou': 0.0, 'test_gqa_loss_giou_0': 0.0, 'test_gqa_loss_giou_1': 0.0, 'test_gqa_loss_giou_2': 0.0, 'test_gqa_loss_giou_3': 0.0, 'test_gqa_loss_giou_4': 0.0, 'test_gqa_accuracy_answer_attr_unscaled': 0.8149443561208267, 'test_gqa_accuracy_answer_cat_unscaled': 0.9180445151033386, 'test_gqa_accuracy_answer_global_unscaled': 0.9894276629570747, 'test_gqa_accuracy_answer_obj_unscaled': 0.9864864864864865, 'test_gqa_accuracy_answer_rel_unscaled': 0.6914944356120827, 'test_gqa_accuracy_answer_total_unscaled': 0.6166931637519872, 'test_gqa_accuracy_answer_type_unscaled': 0.9996820349761526, 'test_gqa_cardinality_error_unscaled': 2.4625596184419716, 'test_gqa_cardinality_error_0_unscaled': 95.7489666136725, 'test_gqa_cardinality_error_1_unscaled': 100.0, 'test_gqa_cardinality_error_2_unscaled': 20.373052464228934, 'test_gqa_cardinality_error_3_unscaled': 31.643481717011127, 'test_gqa_cardinality_error_4_unscaled': 64.79006359300477, 'test_gqa_loss_answer_attr_unscaled': 0.5937237836435415, 'test_gqa_loss_answer_cat_unscaled': 0.40440608162500535, 'test_gqa_loss_answer_global_unscaled': 0.03414623715799758, 'test_gqa_loss_answer_obj_unscaled': 0.03846898033370753, 'test_gqa_loss_answer_rel_unscaled': 1.462530900234227, 'test_gqa_loss_answer_type_unscaled': 0.000580944533996148, 'test_gqa_loss_bbox_unscaled': 0.0, 'test_gqa_loss_bbox_0_unscaled': 0.0, 'test_gqa_loss_bbox_1_unscaled': 0.0, 'test_gqa_loss_bbox_2_unscaled': 0.0, 'test_gqa_loss_bbox_3_unscaled': 0.0, 'test_gqa_loss_bbox_4_unscaled': 0.0, 'test_gqa_loss_ce_unscaled': 2.1158246287386064, 'test_gqa_loss_ce_0_unscaled': 157.7743893561189, 'test_gqa_loss_ce_1_unscaled': 96.71868126752456, 'test_gqa_loss_ce_2_unscaled': 14.313661651126152, 'test_gqa_loss_ce_3_unscaled': 20.294675855682083, 'test_gqa_loss_ce_4_unscaled': 70.58084329565682, 'test_gqa_loss_giou_unscaled': 0.0, 'test_gqa_loss_giou_0_unscaled': 0.0, 'test_gqa_loss_giou_1_unscaled': 0.0, 'test_gqa_loss_giou_2_unscaled': 0.0, 'test_gqa_loss_giou_3_unscaled': 0.0, 'test_gqa_loss_giou_4_unscaled': 0.0, 'test_gqa_coco_eval_bbox': [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0], 'n_parameters': 185847022}

[andics]

Hi there,

I ran into the exact same issue. Could you provide some information on how you solved it since I see you closed this issue? It really would help the community!

Thanks in advance, Andrey