search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
705 stars 53 forks source link

Overridden the async function `init` in the output file `_tiktoken.js` #11

Closed aliuq closed 1 year ago

aliuq commented 1 year ago

Sorry, I'm not very familiar with Rust. Relation target is web and no-modules

image

dqbd commented 1 year ago

Hello, I assume the intent is to load tiktoken library in a different non-Node / Bun environment. If that is the case, #12 should most likely address your issues.

aliuq commented 1 year ago

Hello

I found that a new init.js file has been added to the submission history.

I am using it in Cloudflare Workers and importing it from the dist/web. I noticed that you are planning to support Cloudflare Workers. However, I have tried several ways to import the file, but none of them have worked well. When I tried to import it using the import statement, I received a script_too_large error. When I tried to use a fetch request, I received a WebAssembly.instantiate(): BufferSource argument is empty error.

dqbd commented 1 year ago

Regarding BufferSource argument is empty: It might be possible that you've forgotten to obtain the arrayBuffer from the fetch request.

Regardless, AFAIK Cloudflare Workers only support 1 MB in the free tier, whereas the WASM binary with the embeddings is 1.7 MB gzipped. Currently still investigating the API, but if you wish, you can build a version of tiktoken without inlined ranks by passing --no-default-features in wasm-pack. This will be addressed in the next upcoming version.

dqbd commented 1 year ago

Hello, a new version (1.0.0) has been released, which should introduce a much smaller WASM binary, especially useful for edge runtimes @aliuq