tokenizer not working on browser

anthropics / anthropic-tokenizer-typescript

MIT License

52 stars 4 forks source link

tokenizer not working on browser #7

Open mohsinulhaq-deshaw opened 1 year ago

mohsinulhaq-deshaw commented 1 year ago

Tried this on a CRA app with WASM enabled.

We could easily fix this by adding an esm build besides the cjs. For now, I had to copy over the token count logic to my app and import the json from the package.