is there a javascript version for tokenizers

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

https://huggingface.co/docs/tokenizers

Apache License 2.0

8.69k stars 747 forks source link

is there a javascript version for tokenizers #1387

Closed Zwe1 closed 6 months ago

Zwe1 commented 8 months ago

we want to count token length of user inputs in web runtime, so it's important for us to provide a javascript version for tokenizers, typescript will be better. All my sincerely thanks for supporting~

ArthurZucker commented 7 months ago

Did you check the node bindings? ( npm install tokenizers@latest) should be able to use this no?

github-actions[bot] commented 6 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.