relative_position_bias_table initialization - Githubissues

rishigami / Swin-Transformer-TF

Tensorflow implementation of Swin Transformer model.

Apache License 2.0

198 stars 46 forks source link

relative_position_bias_table initialization #11

Closed gathierry closed 2 years ago

gathierry commented 2 years ago

Hi, In the official code, relative_position_bias_table is initialized in a truncated normal distribution. Is that part missing in this repo?

Official code: https://github.com/microsoft/Swin-Transformer/blob/6bbd83ca617db8480b2fb9b335c476ffaf5afb1a/models/swin_transformer.py#L110

This implem https://github.com/rishigami/Swin-Transformer-TF/blob/8986ca7b0e1f984437db2d8f17e0ecd87fadcd4f/swintransformer/model.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L70

rishigami commented 2 years ago

I forgot to implement that. thx! btw when using pretrained weights, we can ignore the initialization things.