Problems on training MaPLE on ImageNet for cross-dataset transfer

[bighuang624]

Hi, dear authors. Congratulations on the acceptance of this excellent paper to CVPR 2023 and thank you for open-sourcing the code. I'm following RUN.md to train MaPLE on ImageNet for cross-dataset transfer, where I found several problems:

1. The training command seems wrong. I think it should be

   # seed=1 
   bash scripts/maple/xd_train_maple.sh imagenet 1

instead of the current given one

# seed=1 
bash scripts/maple/xd_train_maple.sh 1

2. Although I changed the training command, I met the following error during training:

Obviously, this error is caused by that a ground-truth label is out of the range of the pre-fixed prediction label space. Therefore, I printed the shape of the logits as well as the provided labels, and found

label: tensor([998, 745, 675, 980], device='cuda:0')
logits.shape: torch.Size([4, 991])    # So a 998 label causes the error

Although I think I found the reason why the error occurred, I'm currently not sure how to fix it, since I'm not too familiar with the whole project. Therefore, I hope you can provide a solution. Also, I likewise found that in the base-to-novel experiment, the logits obtained by training on ImageNet were [4, 496]. This did not introduce an error, but I would like to understand where or by what method these two numbers, 991 and 496 , were set.

Hope to get your reply soon. Much thanks.

[bighuang624]

Hi. I have done the same operation with CoOpOp's code, and found the same error. So I suspect it's my own problem with my ImageNet dataset. I got the dataset by filtering the ImageNet-21k with the classnames.txt, since I didn't know which concrete dataset should I download from the ImageNet official website (seems that they don't offer a dataset version named ImageNet-1k). Do you have any suggestions about downloading the appropriate dataset or anything else helpful for fixing the problem?

P.S. I have another question. How can I get the results of ImageNet (i.e., Source) in cross-dataset transfer and domain generalization experiments? I have tried bash scripts/maple/reproduce_maple_xd.sh imagenet 1 /path/to/imagenet/weights/folder with the provided pre-trained imagenet model weights for MaPLe, but it produced a pretty low accuracy of 11.9%. So I guess this is not the right command. Looking forward to a reply.

[muzairkhattak]

Hi @bighuang624,

Thank you for showing interest in MaPLe.

Regarding the ImageNet dataset, you can download the corresponding ImageNet dataset from Kaggle at this link.

P.S. I have another question. How can I get the results of ImageNet (i.e., Source) in cross-dataset transfer and domain generalization experiments? I have tried bash scripts/maple/reproduce_maple_xd.sh imagenet 1 /path/to/imagenet/weights/folder with the provided pre-trained imagenet model weights for MaPLe, but it produced a pretty low accuracy of 11.9%. So I guess this is not the right command. Looking forward to a reply.

I have verified and the command seems to be working fine for me. I think you will need to first download the dataset correctly and then repeat the evaluation again.

As you are trying to reproduce the results on ImageNet dataset, I believe it will work fine when you prepare the dataset correctly.

Kindly let me know in-case there is any other query.

Thank you and kind regards.

[bighuang624]

@muzairkhattak Hi! Much thanks for your help. I have successfully resolved it after downloading the dataset from the given link, and reproduced the cross-dataset transfer/domain generalization results of ImageNet. Thank you and kind regards.

[muzairkhattak]

We are glad to hear that! Kind regards.