waylaidwanderer
commented
1 year ago Here is my function used to count tokens:
/**
* Algorithm adapted from "6. Counting tokens for chat API calls" of
* https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
* @param {*[]} messages
*/
static getTokenCountForMessages(messages) {
// Get the encoding tokenizer
const tokenizer = get_encoding('cl100k_base');
// Map each message to the number of tokens it contains
const messageTokenCounts = messages.map((message) => {
// Map each property of the message to the number of tokens it contains
const propertyTokenCounts = Object.entries(message).map(([key, value]) => {
// Count the number of tokens in the property value
const numTokens = tokenizer.encode(value).length;
// Subtract 1 token if the property key is 'name'
const adjustment = (key === 'name') ? 1 : 0;
return numTokens - adjustment;
});
// Sum the number of tokens in all properties and add 4 for metadata
return propertyTokenCounts.reduce((a, b) => a + b, 4);
});
// Sum the number of tokens in all messages and add 2 for metadata
return messageTokenCounts.reduce((a, b) => a + b, 2);
}
dqbd
I'm getting this error sometimes but there aren't any weird unicode characters in the input or anything else weird, just regular English text.
Trying to continue using it gives me this error: