search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.68k stars 746 forks source link

Unsupported platform for tokenizers #1418

Closed KolbySisk closed 5 months ago

KolbySisk commented 6 months ago

When trying to use tokenizers I get this error: Cannot find module 'tokenizers-win32-x64-msvc'

After install the package it works fine locally, but when I try to deploy to Vercel I get this error: 

npm ERR! notsup Unsupported platform for tokenizers-win32-x64-msvc@0.13.4-rc1: wanted {"os":"win32","cpu":"x64"} (current: {"os":"linux","cpu":"x64"})

I'm guessing I need to install tokenizers-linux-x64-gnu on Vercel, but I don't see anyone else reporting this problem so I'm wondering if I'm doing something wrong here.

ArthurZucker commented 6 months ago

Your understanding is correct; you'll need to install the appropriate package for the Linux environment, such as tokenizers-linux-x64-gnu. (for the vercel env)

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.