javirandor / anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream
MIT License
115 stars 9 forks source link

Add a License #3

Closed andyliucode closed 4 months ago

andyliucode commented 7 months ago

This is amazing work, and since the cost of using Claude 3 is measured in dollars per token (or cents per million tokens), I can imagine many people could benefit from using your code to estimate the cost of Claude 3 without sending any data to Anthropic.

However, this is currently impossible because your work is unlicensed! As such, any potential users face uncertain legal risk if they want to use your code without your explicit permission.

I suggest that you attach a license. Github has documentation on choosing the right license, and in fact they have a nice writeup on choosing the right license here. My recommendation is the MIT license, since that is what openai/tiktoken uses.