search

lucidrains / lambda-networks

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute
MIT License
1.53k stars 157 forks source link

Positional Encoding #29

Open JeremyCJM opened 3 years ago

JeremyCJM commented 3 years ago

Why the positional encoding is random?

johndpope commented 3 years ago

maybe ask @khanrc https://github.com/lucidrains/lambda-networks/commit/1ac51c0ce52de0ba78e2ab97152d2bafe2543b95#diff-6d46aff567baa42dfd4c6b8d2f5c956d3ea84b2811ca663ea67159e323344ccb

khanrc commented 3 years ago

It is learnable positional embedding, which is initialized randomly and learned during training. Please refer to the paper.