Rotary positional embeddings

cfoster0 / CLAP

Contrastive Language-Audio Pretraining

BSD 3-Clause "New" or "Revised" License

87 stars 4 forks source link

Closed cfoster0 closed 3 years ago

cfoster0 commented 3 years ago

Would close #12.

cfoster0 commented 3 years ago

Right now, this includes the CLS token. Should probably exclude it from posembs.

lucidrains commented 3 years ago

Looks good! The only other thing is that sin and cos can be calculated once at the beginning, and passed to each attention module on forward

cfoster0 commented 3 years ago

Ah right! Smart thinking.

lucidrains commented 3 years ago

Lgtm! 💯