search

muzairkhattak / multimodal-prompt-learning

[CVPR 2023] Official repository of paper titled "MaPLe: Multi-modal Prompt Learning".
https://muzairkhattak.github.io/multimodal-prompt-learning/
MIT License
578 stars 43 forks source link

Can NOT reproduce the reported results of EuroSAT. #28

Closed JimZAI closed 1 year ago

JimZAI commented 1 year ago

Hi, thank you for the nice work.

After several more attempts, we were unable to reproduce the results of the EuroSAT dataset in the paper using the same seeds and other hyper-parameters you specified.

Please explain the reasons for this.

Best,

JimZAI commented 1 year ago

Concretely, Ours (Yours): Base | Novel | H 90.03 (94.07) | 73.23 (73.23) | 80.77 (82.35)

muzairkhattak commented 1 year ago

Hi @JimZAI,

Thank you for showing interest in our work!

Regarding your query, can you please verify the checks below:

1) Are you using the same packages such as PyTorch version as mentioned in our INSTALL.md instruction file? 2) Please refer to the provided pre-trained weights and particularly its log files, using which you can track down and analyze our training. The log file for our eurosat training can be found at this link.

Also note that sometimes the results can fluctuate slightly on different machines even when using same configurations. I have observed this trend many times.

I hope that would be helpful. Kindly let us know if this resolves your issue!

Thank you and kind regards.

JimZAI commented 1 year ago

Thanks for your response!

I'll try it.

PoopBear1 commented 3 months ago

@muzairkhattak Sorry to reopen this issue again.

Since I find the same promblem that I cannot reproduce the same result with same code. And I just check the link your provided, it seems shows the trainer is CoOOP instead of MaPLe?

Any comments would be appreciated!

image

muzairkhattak commented 3 months ago

Hi @PoopBear1,

Thank you for reaching out to us!

Regarding the log files provided, kindly note that they show the training and testing history for the MaPLe architecture. The MaPLe framework during our experiments was using CoCoOp trainer, due to which it shows CoCoOp trainer.

However, the results must not vary because of the change of trainer as both trainer trains the model identically. We could reproduce the same results with the same code provided in this repository.

The fluctuations in results might occur due to changes in GPU machines, as we have also observed the same. Additionally, the environment and package versions should be the same as those provided in the INSTALL.md file.

I hope that would be helpful.

Thank you and kind regards!