search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.76k stars 801 forks source link

I'm consistently getting this `wasm RuntimeError: unreachable` for large files #150

Closed manikanta-kondeti closed 1 year ago

manikanta-kondeti commented 1 year ago

wasm://wasm/00b5f812:1

RuntimeError: unreachable at wasm://wasm/00b5f812:wasm-function[563]:0x6a72a at wasm://wasm/00b5f812:wasm-function[665]:0x6fd7a at wasm://wasm/00b5f812:wasm-function[756]:0x70f7f at wasm://wasm/00b5f812:wasm-function[237]:0x5c43a at wasm://wasm/00b5f812:wasm-function[200]:0x4db89 at wasm://wasm/00b5f812:wasm-function[34]:0x1f78a at wasm://wasm/00b5f812:wasm-function[159]:0x48dc3 at Tiktoken.encode (/node_modules/@dqbd/tiktoken/tiktoken_bg.cjs:262:18)

This is the usage

    const encoding = get_encoding('cl100k_base');
    totalTokenCount += encoding.encode(prompt).length;
hauntsaninja commented 1 year ago

dqbd/tiktoken

That's an unofficial fork. Please report to that repo.