Support more rotary embedding options for Llama

elixir-nx / bumblebee

Pre-trained Neural Network models in Axon (+ 🤗 Models integration)

Apache License 2.0

1.27k stars 90 forks source link

Closed jonatanklosko closed 7 months ago

jonatanklosko commented 7 months ago

Closes #278.

Added :rotary_embedding_base to the Llama config, which we already had implemented, and added support for rotary embedding scaling.