A minor detail about positional embedding

facebookresearch / esm

Evolutionary Scale Modeling (esm): Pretrained language models for proteins

MIT License

3.16k stars 627 forks source link

A minor detail about positional embedding #132

Closed Yijia-Xiao closed 2 years ago

Yijia-Xiao commented 3 years ago

Hi, I found an interesting phenomenon in the process of using your MSA Transformer model. In the LearnedPositionalEmbedding class, you added padding_idx to positions (kind of like offset). I wonder why padding_idx is added and is this addition necessary? Thank you!