search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.69k stars 747 forks source link

cannot install with yarn & missing module in npm #1365

Closed MaelAbgrall closed 6 months ago

MaelAbgrall commented 9 months ago

Hello

I'm trying to use the node binding for tokenizers, and, as in the title, yarn fails to install the package. with yarn 1

Error: Couldn't find package "tokenizers-darwin-arm64@0.13.3" required by "tokenizers@latest" on the "npm" registry.
Error: Couldn't find package "tokenizers-win32-ia32-msvc@0.13.3" required by "tokenizers@latest" on the "npm" registry.

with yarn 3

➤ YN0000: ┌ Resolution step
➤ YN0035: │ tokenizers-android-arm-eabi@npm:0.13.3: The remote server failed to provide the requested resource
➤ YN0035: │   Response Code: 404 (Not Found)
➤ YN0035: │   Request Method: GET
➤ YN0035: │   Request URL: https://registry.yarnpkg.com/tokenizers-android-arm-eabi
➤ YN0000: └ Completed in 2s 274ms
➤ YN0000: Failed with errors in 2s 277ms

Similarly, I tried to install it with npm, but then I'm getting the following error while running my code: Error: Cannot find module 'tokenizers-linux-x64-gnu'

Can someone help?

thanks

berrazuriz1 commented 8 months ago

Did you fix it?

MaelAbgrall commented 8 months ago

nope, I gave up

guotingchao commented 7 months ago

same +1

ArthurZucker commented 7 months ago

cc @Narsil I have no idea about this one 😅

Narsil commented 7 months ago

Oh we didn't release anything on NPM since the move the napi-rs because it changed the whole release cycle.

cd bindings/node
yarn build

Should already be something to workaround it.

@ArthurZucker Do you want to mess with node builds ? :) I'd like to find the time for it tbh.

ArthurZucker commented 7 months ago

😈 wish I had the time as well i'll keep it in mind