Do we need to use the inflated swin2d parameters to initialise swin3d? Won't initialising using the converted pre-trained pytorch weight be enough?

[shoaib6174]

@sayakpaul Please have a look.

The Video Swin Transformer (swin3d) initialises weights in two ways-

1. Using the pre-trained swin3d weights
2. Using the inflated weights of Swin Transformer (swin2d) models.

Code Link: https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/mmaction/models/backbones/swin_transformer.py#L563

Our target is to convert the pre-trained pytorch weight to tf2 not training the model. So do we need to initialise the model using inflated swin2d weights? Won't initialising the models using the converted pre-trained weight be enough?

If initialising the model with inflated swin 2d weights needed, then should we use the original pytorch weights of swin2d models or the converted tf2 weights available on tf-hub?

[sayakpaul]

Let's start with the 3D variant.