Some question about inkd_loss?

[kleinzcy]

Thanks your work, in your code, I notice a inkd_loss which not mentioned(or maybe I have not found it.). I am curious about the function of inkd_loss, and the use of Imagenet1K dataset.

In addition, if I understand correctly, features[1] in line methods/contrastive_training/common.py 351 is a global prompt, and features[1] is different for different samples due to the norm operation and dropout. I have not figure out what's meaning of constrastive loss defined on features[1]?

I would very appreciate it if you can help me.

[sheng-eatamath]

Hi, thanks for your interest in our work. We actually mentioned the INKD loss in the implementation details section in appendix. But we observe there is no significant effect of this loss on medium and large datasets. Furthermore, this loss is included for only first few epoch training in the first stage. Therefore, the effect should be minor. Our initial motivation is to counteract the sensitiveness of randomly initialized visual prompts w.r.t. the performance. Empirically, if we remove this loss, sometimes the performance can improve a little. Thus, we suggest that if you are training on a dataset of a medium or large size, you can directly remove it. We propose discriminative prompt regularization to improve the semantic discriminativeness of the prompt-adapted backbone, which includes additional task-related supervision on the ensembled prompt features. Please look into section 3.2 in our paper. If you have further question, please let us know. Thank you.

[DevinCheung]

@sheng-eatamath I noticed for the INKD loss in contrastive_training_1.py, the 'imagenet' dataset is used as a fixed auxiliary dataset (Line 602). I am wondering if this means only for the experiment on imagenet, this INKD loss is used, or for all experiments on every dataset, imagenet is always used as auxiliary dataset? The latter does not make sense (maybe my understanding is wrong).

[sheng-eatamath]

@DevinCheung Thanks for your good comments. We do compute INKD over all the datasets to keep our implementation consistent. However, several points needed to be clarified: Firstly, notice that we only compute INKD loss for 5 epochs with linear-decay weights to zero for only 1st stage training, which is small and just for counteracting the negative effects of prompt initialization. Thus the positive effect of this loss on the final performance is small/minor, according to our results. Secondly, as for the motivation, we observe this loss can add minor benefits to the first stage performance on small datasets, i.e., CUB-200. In practice, we can use any arbitrary dataset including the downstream dataset itself to do the distillation. Lastly, GCD assumes the Imagenet-1k pretrained DINO, which also assumes the whole ImageNet without labels. Our work follows this setup.