search

javirandor / anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream
MIT License
115 stars 9 forks source link

Project Broken #5

Closed honey-tree closed 3 months ago

honey-tree commented 3 months ago

The idea is simple. Ask Claude to repeat some text and observe how the generation is streamed through the network. It turns out that Anthropic serves one token at a time!

python anthropic_tokenizer.py --text "Good good very berry good."
Tokens: ['Goo', 'd good very berry goo', 'd.']
Number of text tokens: 3
Total tokens usage (as of API): 9

Verifiably no longer the case. Everyone go home, anthropic has changed streaming behavior and this whole thing is now broken.

javirandor commented 3 months ago

Thanks for sharing! Several people have reported this behavior. It looks like Anthropic patched this. However, the strategy of limiting the output tokens to 1 still works for people who really want to find out some specific tokenization.