tokenizer not working on browser

anthropics / anthropic-tokenizer-typescript

MIT License

47 stars 1 forks source link

tokenizer not working on browser #7

Open mohsinulhaq-deshaw opened 11 months ago

mohsinulhaq-deshaw commented 11 months ago

Tried this on a CRA app with WASM enabled.

We could easily fix this by adding an esm build besides the cjs. For now, I had to copy over the token count logic to my app and import the json from the package.