search

chinhsuanwu / coatnet-pytorch

A PyTorch implementation of "CoAtNet: Marrying Convolution and Attention for All Data Sizes"
https://arxiv.org/abs/2106.04803
MIT License
370 stars 67 forks source link

aboult attention model #2

Closed mlxu995 closed 3 years ago

mlxu995 commented 3 years ago

屏幕截图 2021-10-25 222552 Hi, I noticed that the value of self.relative_bias_table is always all 0, then the following: relative_bias = self.relative_bias_table.gather( 0, self.relative_index.repeat(1, self.heads)) is actually meaningless (it is all 0)? Thanks!

chinhsuanwu commented 3 years ago

Hi @mlxu995

As you can see, self.relative_bias_table is a nn.Parameter initialized with zeros that can be learned through training. So it will not be a constant value once you train it.

img

mlxu995 commented 3 years ago

Thank you very much! @chinhsuanwu