search

Haochen-Wang409 / HPM

[CVPR'23] Hard Patches Mining for Masked Image Modeling
https://arxiv.org/pdf/2304.05919.pdf
Apache License 2.0
88 stars 7 forks source link

Request for certain experimental matters #5

Closed zachary19889 closed 1 year ago

zachary19889 commented 1 year ago

Hi, thanks for your great work, I have some experimental matters here.

  1. For the ViT-Base HPM pre-training config, the paper only provides a config for 200 epochs, and I'm not sure whether the pretrain_base.sh in this repository is for 800 epochs or not. Could you provide config for 800 and 1600 epochs?
  2. Why is the optimizer different from the original LARS optimizer in MAE for the linear_probing experiments?
  3. Have you tested different batch sizes during pre-training, Does batch_size impact the model performance significantly? Looking forward to hearing back from you soon. Thank you!
rayleizhu commented 1 year ago

I have the same questions, especially 1&3

Haochen-Wang409 commented 1 year ago

Sorry for the late response.

  1. The provided script is the configuration of 800 epochs of pre-training. This configuration is almost borrowed from UM-MAE.
  2. We have tried the LARS optimizer but we found it performed extremely badly (around 10% Top-1). It might be because we do not use a very large batch size during linear probing.
  3. Batch size will not impact significantly.