Open TranquilMarmot opened 4 months ago
I came across this today. It's preventing me from deploying my NodeJs API. As far as I'm aware, I didn't update this package and yet the error has come about out of thin air. Therefore, this may be something to do with the Google Cloud Functions environment in which I'm running the api. Currently assessing older versions of this package to see if the error persists.
This is a duplicate of https://github.com/dqbd/tiktoken/issues/46 but that's closed, and I don't think it should be 😄
Following the instructions here in the README:
https://github.com/dqbd/tiktoken?tab=readme-ov-file#cloudflare-workers
When you try to deploy the function to Cloudflare, it will fail with:
It's happening at:
Using
js-tiktoken
here isn't an option since the package for a Cloudflare worker has to be < 1MiB (1024 KiB) after gzip, so WASM is really the only route to go if you want to run this in a Cloudflare worker.Note; you can use
gpt-tokenizer
in a Cloudflare worker with something like:It seems like the Cloudflare worker instructions should be removed from the README for now, or at the very least the ✅ should be changed to a ❌.