search

Cainier / gpt-tokens

Calculate the token consumption and amount of openai gpt message
MIT License
106 stars 13 forks source link

Tiktoken encodings not re-used between GPTTokens instances (slow performance) #25

Closed lox closed 1 year ago

lox commented 1 year ago

When calling this library many times (for instance using it to split text into parts), it appears the internal encodings aren't re-used and there isn't any way to do it.

The result is it's quite slow:

import { GPTTokens } from 'gpt-tokens'
// import { GPTTokens } from '../src/libs/gptTokens.js'

for (let i = 0; i < 1000; i++) {
  console.time('GPTTokens')
  const usageInfo = new GPTTokens({
    plus: false,
    model: 'gpt-3.5-turbo-0613',
    messages: [
      {
        role: 'user',
        content: 'Hello world',
      },
    ],
  })

  usageInfo.usedTokens
  usageInfo.promptUsedTokens
  usageInfo.completionUsedTokens
  usageInfo.usedUSD
  console.timeEnd('GPTTokens')
}

Returns:

GPTTokens: 332.625ms
GPTTokens: 290.321ms
GPTTokens: 273.416ms
GPTTokens: 264.106ms
GPTTokens: 281.858ms
GPTTokens: 257.714ms
GPTTokens: 280.463ms
GPTTokens: 282.296ms
GPTTokens: 255.335ms
GPTTokens: 274.843ms
GPTTokens: 268.74ms
GPTTokens: 269.419ms
GPTTokens: 279.843ms
GPTTokens: 252.028ms
GPTTokens: 276.782ms
GPTTokens: 283.575ms
GPTTokens: 258.711ms
GPTTokens: 284.372ms

When the encodings are cached in the module:

GPTTokens: 64.708ms
GPTTokens: 1.558ms
GPTTokens: 1.12ms
GPTTokens: 1.114ms
GPTTokens: 0.876ms
GPTTokens: 0.838ms
GPTTokens: 0.954ms
GPTTokens: 0.92ms
GPTTokens: 0.765ms
GPTTokens: 0.84ms
GPTTokens: 0.72ms
GPTTokens: 0.789ms
GPTTokens: 0.822ms
GPTTokens: 0.782ms
GPTTokens: 0.78ms
GPTTokens: 0.737ms
GPTTokens: 0.746ms
lox commented 1 year ago

The good news is that memory usage is reclaimed!

Cainier commented 1 year ago

Thanks for improving this, I'll be pushing v1.1.1