add argument for model; fix the return types - Githubissues

javirandor / anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream

MIT License

109 stars 9 forks source link

add argument for model; fix the return types #2

Closed javirandor closed 6 months ago

javirandor commented 6 months ago

Add an argument to choose the model from the CLI
Fix the return types in src/anthropic_tokenizer.py from None to tuples.