search

vt-vl-lab / iCAN

[BMVC 2018] iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
https://gaochen315.github.io/iCAN/
MIT License
259 stars 60 forks source link

Substandard performance without postprocessing. #23

Open MayankSingal opened 5 years ago

MayankSingal commented 5 years ago

Good Evening,

There seems to be lot of rule based post processing being done after getting the model outputs. (Forcing prediction to be snowboarding if snowboard is present etc.). Can you report the numbers without such post processing being applied? Evaluating the model after removing post-processing is giving me very bad results, so I'm not sure if I'm doing it properly. I couldn't find any mention of such numbers in the BMVC paper.

Thank you!

gaochen315 commented 5 years ago

Thanks for the question.

We follow the processing steps by prior work (Visual Semantic Role Labeling, Detecting and Recognizing Human-Object Interactions) to remove prediction scores for certain action-object pairs. You can see in the eq2 of Visual Semantic Role Labeling, C is predefined. Thus, we know the object categories that are related to a specific action class.

You can simply add a --prior_flag 0 flag to the Test_ResNet_VCOCO.py, i.e. using the following command 

python tools/Test_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000 --prior_flag 0

to obtain the mAP without any post-processing.

MayankSingal commented 5 years ago

Thank you for the quick response!

I ran the testing pipeline with the above command using prior_flag = 0.

On the pre-trained model provided, the mAP values obtained with prior flag = 0 are 2.38(Scenario 1) and 4.81(Scenario 2). Is such a huge drop expected? Compared to the reported mAP of 45, these are very surprising numbers.

Is the contribution of this prior knowledge 'C' so significant?

Thank you!

MayankSingal commented 5 years ago

Good Morning, Any information on this?

Thank you!

gaochen315 commented 5 years ago

As I recall, we will lose around 3mAP without post-processing. The mAP should be around 42.

I will check and see where is the problem.

ZHUXUHAN commented 5 years ago

Good Morning, Any information on this?

Thank you!

hi, man,do you know the metric for hico-det

Good Morning, Any information on this?

Thank you!

hi, man did you test the vcoco or the hico-det datasets, and how did you test the model, I know just needs to test hico-det model, if you know, can you help me? oh, thank you very much.