Integrate Axial-RoPE - Githubissues

OpenBioML / protein-lm-scaling

Other

54 stars 15 forks source link

Integrate Axial-RoPE #55

Closed pascalnotin closed 9 months ago

pascalnotin commented 9 months ago

Integrate axial RoPE. Implementation of this position encoding scheme can be found here: https://github.com/crowsonkb/k-diffusion/blob/8d81974289d671d7e3fd3a25f11ef3ddf3c1f195/k_diffusion/models/axial_rope.py#L89

talkhanz commented 9 months ago

/take

pascalnotin commented 9 months ago

Closing this issue as we will be working with single axis data input only. May reopen later if we end up doing attention across residues and retrieved sequences as in MSA Transformer.