search

javirandor / anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream
MIT License
109 stars 9 forks source link

Easier and cheaper way to accomplish this #4

Closed stri8ed closed 4 months ago

stri8ed commented 4 months ago

Edit: Disregard. This only for counting the number of tokens.