search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
648 stars 49 forks source link

Memory access out of bounds #84

Open maxdeichmann opened 5 months ago

maxdeichmann commented 5 months ago

I am using tiktoken in my project here: https://github.com/langfuse/langfuse/blob/250a5b03d2068774151757d8a0b33115a56243e2/src/features/ingest/lib/usage.ts#L6

As advised in the docs, i always free up the encoder, after usage. Sometimes, i use the same encoder multiple times without freeing it in between encode() function calls. Is this an issue?

I am running a nextjs project, and configured everything as required:https://github.com/langfuse/langfuse/blob/250a5b03d2068774151757d8a0b33115a56243e2/next.config.mjs#L60

My exact exception is the following:

RuntimeError: memory access out of bounds
    at wasm://wasm/00c6e182:wasm-function[132]:0x70bf8
    at wasm://wasm/00c6e182:wasm-function[982]:0xa8c44
    at passStringToWasm0 (/var/task/node_modules/tiktoken/tiktoken_bg.cjs:80:15)
    at Tiktoken.encode (/var/task/node_modules/tiktoken/tiktoken_bg.cjs:269:26)
    at /var/task/.next/server/chunks/292.js:1:4348
    at Array.forEach (<anonymous>)
    at /var/task/.next/server/chunks/292.js:1:4316
    at Array.forEach (<anonymous>)
    at e (/var/task/.next/server/chunks/292.js:1:4284)
    at e (/var/task/.next/server/chunks/292.js:1:4040)
maxdeichmann commented 5 months ago

@dqbd your help would be highly appreciated.

Luke-Tan commented 4 months ago

hi @maxdeichmann , did you ever resolve this issue?

maxdeichmann commented 4 months ago

I switched to TikToken-JS which works much better