Support transformers-based tokenizers

[darknoon]

• [x] Abstract models & tokenizers out to allow loading transformers tokenizers via @xenova/transformers
• [x] Fix error with case in the model selector ← I had to use a nasty hack that recreates the callbacks every render, but it works
• [x] Fix error with loading model .json from transformers ← we need to auth against the HF api to load tokenizers for gated models. Maybe we should do a proxy, or republish just the tokenizers somewhere? thoughts
• [x] Fix segment calculation—have either been missing whitespace characters or getting extra junk :/ Needs another pass.
• [x] Fix llama3

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
tiktokenizer	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 25, 2024 11:03pm