Use tokenizer_config.json as the primary source of metadata

elixir-nx / bumblebee

Pre-trained Neural Network models in Axon (+ 🤗 Models integration)

Apache License 2.0

1.27k stars 90 forks source link

Closed jonatanklosko closed 6 months ago

jonatanklosko commented 7 months ago

See https://github.com/elixir-nx/bumblebee/issues/278#issuecomment-1808397084. We still want to fallback to the other files for compatibility with old checkpoins.