Question for the zero-shot method evaluation on swig dataset

[ChelsieLei]

Hi @ltttpku,

Thanks for your nice work! I saw in your supplementary materials, you provided the results of the HOICLIP model trained on HICO-DET and tested on swig. However, HOICLIP requires the pre-defined HOI classes to obtain the verb representation (shown in Fig. 4 in HOICLIP paper), which is used in the HOI prediction. How do you deal with this problem? Thanks a lot!

[ltttpku]

Thanks for your interest in our work! We provide the model with predefined HOI classes during inference.

[ChelsieLei]

Hi, thank you for your reply. HOICLIP has a verb classifier that requires the pre-defined unseen classes in training. However, based on the open-vocabulary setting, the unseen classes are only available after training and before inference. Thus, I wonder how you solve this problem when testing HICODET on the SWIG dataset which has unseen classes after the HOICLIP training.

[ltttpku]

I remember that I trained HOICLIP on the HICO-DET dataset, using all HOI classes from HICO-DET by default. During testing, I replaced the embeddings with those from SWIG-HOI.

[ChelsieLei]

Hi, I mean this module in red circle requires the unseen class information. How do you use the verb classifier? The code of the verb implementation is here [https://github.com/Artanic30/HOICLIP/blob/main/models/models_hoiclip/hoiclip.py#L202]

[ltttpku]

The verb classifier was discarded due to the lack of data for visual semantic arithmetic.

[ChelsieLei]

Thanks for your information!