search

JiazuoYu / MoE-Adapters4CL

Code for paper "Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters" CVPR2024
119 stars 6 forks source link

Reproduce Results #12

Closed Areeb2735 closed 3 weeks ago

Areeb2735 commented 1 month ago

Hi.

I am unable to reproduce the results for the MDTL setting. I am running this bash file, srcipts/train/train_full_shot_router11_experts22_1000iters.sh. Am I missing something? The results are noticeably different. I am comparing with table 9 from the paper. Please do let me know.

Thank you.

JiazuoYu commented 1 month ago

Hi,

Could you please provide further settings and results? In addition, you can check whether the DDAS training is correct. You can use the ckpts we provide to check the problems of the two parts of DDAS or MoE-Adapter that you reproduced.

Areeb2735 commented 1 month ago

Screenshot 2024-07-17 at 11 13 05 AM

The results are slightly lower than those in Table 9. Also, the zero-shot on Cifar100 and EuroSAT (in the first two rows) is much less than what you reported. The dataset is loading correctly because the ckpt you provided is able to reproduce the results. Please find attached the screenshot of the results.

I train using the default setting, bash srcipts/train/train_full_shot_router11_experts22_1000iters.sh

Thank you in advance.

JiazuoYu commented 1 month ago

Hi,

Apologies for the delayed response. We reran the experiments and found the results still similar to those in the paper. The retrained and tested results are as follows(Note: The result matrix needs to be transposed):

截屏2024-07-18 11 00 21

Upon careful review, we discovered an error in the code organization due to incorrect copying. You can refer to #13, and we have updated the repository accordingly.

We deeply regret any inconvenience caused by our oversight. Thank you.

Areeb2735 commented 1 month ago

Thank you for your response. I have corrected the code, as you said. Please find attached the results table. The Results are different from what you are reporting. For instance, the first row gives me 53.53%, while it's 50.08% for you. Also, the zero-shot on cigar and erurosat is too different. I have also attached the arguments list when training the experts and routers.

image image

Thank you again.

JiazuoYu commented 1 month ago

For zeroshot performance, you can adjust the different thresholds to check the results of zeroshot.

Regarding the fluctuation of results, we think it is inevitable.