search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.69k stars 747 forks source link

Error: Cannot find module 'tokenizers-linux-x64-musl' #1384

Closed Madnex closed 5 months ago

Madnex commented 8 months ago

Hi! I get the following error when I try to load the tokenizers: Error: Cannot find module '@anush008/tokenizers-linux-x64-musl'. I am using a docker container with node:18-alpine.

I peeked at the index.js file and saw that the isMusl check is for Node 10? What exactly is it checking? And why does the corresponding module not exist anymore?

See also the issue here.

github-actions[bot] commented 7 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

ArthurZucker commented 7 months ago

Not sure if the package was properly installed or not and would need a full traceback + reproducer otherwise

github-actions[bot] commented 6 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

madhurjya-acko commented 3 months ago

@Madnex were you able to resolve this in your case? I am having the same issue.

Madnex commented 3 months ago

Sorry, no did not manage. And did not continue with that solution :/

ArthurZucker commented 4 weeks ago

@madhurjya-acko this should be fix I reckon depending on the version of tokenizers you are installing + the version of python you have