search

tmokmss / bedrock-pr-reviewer

AI-based Pull Request Summarizer and Reviewer with Chat Capabilities.
MIT License
38 stars 9 forks source link

Token count #4

Open tmokmss opened 11 months ago

tmokmss commented 11 months ago

Currently we're using tiktoken to calculate token count (not changed from original implementation.) It seems it's already providing a good estimation but ideally we should use this. https://github.com/anthropics/anthropic-tokenizer-typescript

tmokmss commented 5 months ago

Note that the above library is no longer exactly accurate on Claude 3.

This package can be used to count tokens for Anthropic's older models. As of the Claude 3 models, this algorithm is no longer accurate, but can be used as a very rough approximation. We suggest that you rely on usage in the response body wherever possible.