search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.69k stars 747 forks source link

Allow tokenizers to use huggingface_hub 0.18.0 #1377

Closed clefourrier closed 8 months ago

clefourrier commented 8 months ago

Please don't delete the associated branch, I (possibly) need it for the open llm leaderboard to install the correct version of hf_hub

HuggingFaceDocBuilderDev commented 8 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.