search

lucidrains / bottleneck-transformer-pytorch

Implementation of Bottleneck Transformer in Pytorch
MIT License
671 stars 82 forks source link

the 2d relative position embedding is not Inductive,maybe the FLOATER embedding is better #9

Open AncientRemember opened 3 years ago

AncientRemember commented 3 years ago

reference from Learning to encode position for transformer with continuous dynamical model in ICML 2020

the relative position embedding which used by botnet is not Inductive,this limit the generalization

it not allow inference with difference image size of which used when traing

lucidrains commented 3 years ago

yea you are right, but I'm just being faithful to the original implementation

Can you drop the link to the paper with the better positional encoding you are referring to?

AncientRemember commented 3 years ago

yea you are right, but I'm just being faithful to the original implementation

Can you drop the link to the paper with the better positional encoding you are referring to the link is updated