search

shkarupa-alex / tfswin

Keras (TensorFlow v2) reimplementation of Swin Transformer V1 and V2 models
MIT License
20 stars 2 forks source link

Pretraining question #4

Open noahcjones opened 6 months ago

noahcjones commented 6 months ago

Hi, does the SwinTransformer v2 do SimMiM pretraining? This is shown in the paper: https://arxiv.org/pdf/2111.09883.pdf

If not, any plans to add/how difficult would it be to port?

shkarupa-alex commented 6 months ago

No, provided weights converted from original V1 and V2 models trained on ImageNet (22k when available, 1k otherwise). If you want to try simmim weights you can modify this script https://github.com/shkarupa-alex/tfswin/blob/master/convert_weights.py to port required weights

noahcjones commented 6 months ago

Oh okay, I ask because I'd like to pretrain on my own, not use the converted weights. I see the example here vit-tensorflow and would like to do something similar with the swinv2.