training under fully supervised and UO zero-shot settings

[PradKalkar]

Hi. Thanks for your work!

Would you please guide me if I have to set up a training script for training EoID under fully supervised setting? What dataset file should I be using for that? Also, if I want to evaluate on UO setting, is there any dataset file in the code which can be directly used? If not, would you please suggest what dataset file needs to be modified to construct the one for UO?

[mrwu-mac]

For fully supervised setting: hico.py is original dataset file of CDN for fully supervised setting. In addition, there is still room for improvement to avoid noise from CLIP logits for the fully supervised setting.

For UO setting: there is not a UO dataset file in our code. It is better to construct UO dataset file based on hico_ua_st_v2.py.

[PradKalkar]

Thanks for the quick reply. I have few more questions -

1. When you say improvement, you are talking about in the context of model, right? And not in the hico.py file?
2. What is the difference between hico_ua_st_v1.py and hico_ua_st_v2.py?

[mrwu-mac]

1. Yes.
2. We update hico_ua_st from v1 to v2, making it more general and clearer.

[PradKalkar]

Thank you so much @mrwu-mac for the answers!

[PradKalkar]

Hi. Just a small question again. What is the difference between the dataset files "hico_uc_base" and "hico_uc_st"? For UC setting, which one to use, could you please tell?

[mrwu-mac]

We have updated train.sh, please uncomment 'EoID UC setting' for UC training.

[PradKalkar]

Thanks!

[PradKalkar]

Hi. Would you kindly confirm whether it is necessary to update hico_ua_eval.py as well for training in UO setting? I personally think that it may not be necessary, but still wanted a confirmation since I am not really sure. I have updated the hico_ua_st_v2.py file as you suggested. So, would it be fine if I pass "--dataset_file hico_ua_st_v2" now into the shell script for UO training without updating the hico_ua_eval.py file?

[mrwu-mac]

No. It is only necessary to update set_ua_hois() in your UO dataset_file. The hico_ua_eval.py is universal for hico unseen setting. We will update the name of the hico_ua_eval.py later.

[PradKalkar]

I think an update to "get_ua" function to return just the empty list is also necessary in the dataset file (hico_ua_st_v2) for UO setting, in addition to the update you suggested, right?

[mrwu-mac]

The get_ua() is used to determine which samples belong to Unseen. So you can update it according to your needs.

[PradKalkar]

While training the model in the UO setting, should we be using the same value for the "pretrained" argument (DETR pretrained weights) as used in train.sh file provided?

[mrwu-mac]

It's a little bit out of our scope. Under the UO setting of the most previous works, the object detector(such as pretrained DETR) has seen all the objects including unseen objects, and the HOI model is only need to combine the actions with unseen objects for UO setting. In my opinion, this is also need to combin the novel object detection into the HOI model.

[PradKalkar]

Thanks for the reply!

[PradKalkar]

Hi @mrwu-mac. I have created a pull-request for the hico_uo_st.py file in the UO setting. Would you please take a look at it and merge it accordingly in your repo if everything is fine? If there are any issues, please let me know. It will be helpful for training and testing EoID in UO setting.

[mrwu-mac]

Sorry, we currently have no plans to merge it, and we don't have additional computing resources to verify it, you may try to verify it yourself.