search

Alpha-VL / ConvMAE

ConvMAE: Masked Convolution Meets Masked Autoencoders
MIT License
477 stars 41 forks source link

ImageNet Evaluation #9

Closed SheldonHS closed 2 years ago

SheldonHS commented 2 years ago

Thanks for sharing the great work. I encountered difficulties in reproducing the evaluation results on FINETUNE.md. My evaluation results are: * Acc@1 1.090 Acc@5 2.188 loss 8.955 Accuracy of the network on the 50000 test images: 1.1% That's obviously too big a gap.

I download the ImageNet-1K following your guidance and prepared the ImageNet-1K following Jasonlee1995. Are there any details I haven't noticed, or any specific requirements for preparing the dataset？

TeleeMa commented 2 years ago

@SheldonHS Could you provide more details about your experiment?

  1. Did you do evaluation with the fine-tuned weights of ours or weights you fine-tuned by yourself.
  2. If you utilized provided fine-tuned weights, make sure that you had downloaded the right model in https://drive.google.com/file/d/19F6vQUlITpzNLvXLKi5NRxRLOmKRxqFi/view.
  3. If you fine-tuned the model by yourself, please provide more training details like the configs.
SheldonHS commented 2 years ago

@TeleeMa Thanks for your reply. I utilized the provided fine-tuned weights. For the sake of insurance, I downloaded and tested it again, but the result is still the same. I experimented strictly following FINETUNE.md, except for the CUDA version. The test program runs on three Tesla V100.

TeleeMa commented 2 years ago

@SheldonHS Then could you provide the evaluation log for me to check?

SheldonHS commented 2 years ago

@TeleeMa Thank you very much for your patience. I've checked many times before opening this issue, so please forgive my bother. Here's the log. ImageNetEvaluation.log

TeleeMa commented 2 years ago

@SheldonHS I downloaded the codes and checkpoint just now and tested the evaluation, here is my log.txt. Also, I tested with different number of gpus and batch sizes, that made no differences to the final results. Therefore, would you mind comparing your log with mine firstly, and any further questions are welcomed. PS. Don't forget check the ImageNet dataset as we use the same format with DeiT and MAE.

SheldonHS commented 2 years ago

@TeleeMa Thanks a lot. It turns out to be the dataset. I tried another way to prepare IN-1k and the results are finally back on track.