Unable to download tokenizer. S3 link seems to be inaccessible

[reevr]

I tried installing the npm package. One of its step includes download of the tokenizer from https://tokenizers-releases.s3.amazonaws.com/node/0.7.0/index-v0.7.0-node-v83-linux-x64-glibc.tar.gz This seems to be inaccessible.

I am using node v12. Can you please help me with this?

[cabernal]

Bumping this issue, getting the same issue:

 ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node-pre-gyp install
npm ERR! 403 status code downloading tarball https://tokenizers-releases.s3.amazonaws.com/node/0.7.0/index-v0.7.0-node-v102-win32-x64-unknown.tar.gz
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using node-pre-gyp@0.14.0
npm ERR! node-pre-gyp info using node@17.1.0 | win32 | x64
npm ERR! node-pre-gyp WARN Using needle for node-pre-gyp https download
npm ERR! node-pre-gyp info check checked for "C:\Users\admin\git\huggingface-qa\node_modules\tokenizers\bin-package\index.node" (not found)
npm ERR! node-pre-gyp http GET https://tokenizers-releases.s3.amazonaws.com/node/0.7.0/index-v0.7.0-node-v102-win32-x64-unknown.tar.gz
npm ERR! node-pre-gyp http 403 https://tokenizers-releases.s3.amazonaws.com/node/0.7.0/index-v0.7.0-node-v102-win32-x64-unknown.tar.gz
npm ERR! node-pre-gyp ERR! install error

[DhairyaBahl]

Any updates with this issue ? I am facing the same issue. Kindly update the package with the new link.

[julien-c]

@SaulLu @Narsil Do you know if Node is still supported as a compilation target from huggingface/tokenizers? Also we might want to transfer this issue to that repo.

[SaulLu]

Unfortunately I don't know either 😞

[Narsil]

tokenizers does support and provide build targets for node, 12 and 14 (not 16 atm iirc linked to us needing to do a big upgrade on the bindings side).

Everything is available through npm install tokenizers normally. Not sure what that old s3 link is.

[creatorrr]

Downgrading to node 12 does seem to work but, as @Narsil mentioned, newer versions don't work. I also get the following deprecation warning: npm WARN deprecated tokenizers@0.12.0: This version has been yanked and won't be supported

Can someone provide an update on this package's future?

Correction: Node 14 also didn't work. Only 12 seems to be working

[Narsil]

@creatorrr ,

PRs are welcome. but updating to new node versions require updating to neon==latest which requires a lot of changes(didn't try yet, but it felt like a rewrite basically).

I am not going to lie, no one is dedicated specifically to tokenizers in general (as it works mostly as intended for transformers which is our biggest user). So spending time rewriting for node is not super high on the prio list right now.

PRs are very welcome if you feel like it !

[gouldcs]

hey is this repo still active? Would love to help get this working again. Trying to get an application working with this and jumping through hoops with Node/Python just to get stuff installed.

[Narsil]

Last commit is from 3 years ago...

[gouldcs]

@Narsil i noticed, but I see activity in issues as recent as last year. Is this package considered completely abandoned at this point?

[Narsil]

Definitely not actively maintained. tokenizers and its node bindings are, but they are in maintenance mode only (until someone wants to help redo the bindings with latest neon)

[julien-c]

we can archive this repo btw maybe

[Narsil]

@Pierrci Are you ok with archiving this ?

[gouldcs]

if this repo is being archived, is there a newer, more reliable open source tool I can use to achieve the same behavior?

[gouldcs]

For those looking for an alternative, I found @tensorflow-models/qna. Does the same thing!