search

lalalune / arcprize

34 stars 4 forks source link

Add RoPE #18

Closed lalalune closed 1 month ago

lalalune commented 1 month ago

This PR integrates xformers RoPE embeddings which should have a better result for our use case than sinusoidal