Closed Areeb2735 closed 3 weeks ago
Hi,
Could you please provide further settings and results? In addition, you can check whether the DDAS training is correct. You can use the ckpts we provide to check the problems of the two parts of DDAS or MoE-Adapter that you reproduced.
The results are slightly lower than those in Table 9. Also, the zero-shot on Cifar100 and EuroSAT (in the first two rows) is much less than what you reported. The dataset is loading correctly because the ckpt you provided is able to reproduce the results. Please find attached the screenshot of the results.
I train using the default setting, bash srcipts/train/train_full_shot_router11_experts22_1000iters.sh
Thank you in advance.
Hi,
Apologies for the delayed response. We reran the experiments and found the results still similar to those in the paper. The retrained and tested results are as follows(Note: The result matrix needs to be transposed):
Upon careful review, we discovered an error in the code organization due to incorrect copying. You can refer to #13, and we have updated the repository accordingly.
We deeply regret any inconvenience caused by our oversight. Thank you.
Thank you for your response. I have corrected the code, as you said. Please find attached the results table. The Results are different from what you are reporting. For instance, the first row gives me 53.53%, while it's 50.08% for you. Also, the zero-shot on cigar and erurosat is too different. I have also attached the arguments list when training the experts and routers.
Thank you again.
For zeroshot performance, you can adjust the different thresholds to check the results of zeroshot.
Regarding the fluctuation of results, we think it is inevitable.
Hi.
I am unable to reproduce the results for the MDTL setting. I am running this bash file, srcipts/train/train_full_shot_router11_experts22_1000iters.sh. Am I missing something? The results are noticeably different. I am comparing with table 9 from the paper. Please do let me know.
Thank you.