search

Zian-Xu / Swin-MAE

Pytorch implementation of Swin MAE https://arxiv.org/abs/2212.13805
62 stars 12 forks source link

The implemente detail? #6

Open s1mple-fake opened 1 year ago

s1mple-fake commented 1 year ago

Excuse me, i want to pretrain a model on the other dataset, but the model i got can't fulfill my demand and got bad performance. My experiment setting is same as you, the epoch is setted as 400, and batchsize is 196 as same as your code. So i want to ask some questions: 1: Does your model use the swin_tiny structure? Did you use Swin_base or more large model? 2: Can you tell me the device that you used in your experiment? My experiment used a RTX 3060 with 12GB memory(out of memory) but i think enlarge the batchsize could make the pretrained model more stable. 3: If you use different structures of Swin Transformer, like Swin_base or Swin_large, would you like to share your pretrained models on the GitHub? There are my question, please forgive my poor English, Thank you!

Zian-Xu commented 1 year ago

Hello, sorry for the late reply. I will answer your questions in turn:

  1. The backbone of the Swin MAE needs to be the same as the downstream task network. Since my downstream task uses the Swin-Unet network, the backbone uses Swin_tiny.
  2. I am using NVIDIA RTX A4000 with 16G memory. In addition, the batch size should be 96 in my code, not 196.
  3. I am not using a bigger swin structure. Since it is pre-trained on a small medical image dataset, it is less generalizable to other tasks than a pre-trained model on a large dataset like ImageNet. So I see little use in making the pre-trained model publicly available. However, I can make the pre-trained model available if you still need it.