Update tokenizers requirement from <0.14,>=0.10 to >=0.10,<0.15

Updates the requirements on tokenizers to permit the latest version.

Release notes

v0.14.0

⚠️ Reworks the release pipeline. Other breaking changes ⚠️ :

#1335, AddedToken is reworked, is_special_token rename to special for consistency

feature http is now OFF by default, and depends on hf-hub instead of cached_path (updated cache directory, better sync implementation)

Removed SSL link on the python package, calling huggingface_hub directly instead.

New dependency : huggingface_hub (while we deprecate Tokenizer.from_pretrained(...) to Tokenizer.from_file(hugginngface_hub.hf_hub_download(MODEL_ID, "tokenizer.json")

What's Changed

Fix conda release by @ArthurZucker in huggingface/tokenizers#1211

Fix node release by @ArthurZucker in huggingface/tokenizers#1212

Printing warning to stderr. by @Narsil in huggingface/tokenizers#1222

Fixing padding_left sequence_ids. by @Narsil in huggingface/tokenizers#1233

Use LTO for release and benchmark builds by @csko in huggingface/tokenizers#1157

fix unigram.rs test_sample() by @chris-ha458 in huggingface/tokenizers#1244

implement a simple max_sentencepiece_length into BPE by @chris-ha458 in huggingface/tokenizers#1228

Makes decode and decode_batch work on borrowed content. by @mfuntowicz in huggingface/tokenizers#1251

Update all GH Actions with dependency on actions/checkout by @mfuntowicz in huggingface/tokenizers#1256

Parallelize unigram trainer by @mishig25 in huggingface/tokenizers#976

Update unigram/trainer.rs by @chris-ha458 in huggingface/tokenizers#1257

Fixing broken link. by @Narsil in huggingface/tokenizers#1268

fix documentation regarding regex by @chris-ha458 in huggingface/tokenizers#1264

Update Cargo.toml by @chris-ha458 in huggingface/tokenizers#1266

Update README.md - Broken link by @sbhavani in huggingface/tokenizers#1272

[doc build] Use secrets by @mishig25 in huggingface/tokenizers#1273

Improve error for truncation with too high stride by @boyleconnor in huggingface/tokenizers#1275

Add unigram bytefallback by @ArthurZucker in huggingface/tokenizers#1217

revise type specification by @hiroshi-matsuda-rit in huggingface/tokenizers#1289

Bump tough-cookie from 4.0.0 to 4.1.3 in /bindings/node by @dependabot in huggingface/tokenizers#1291

Update path name: master -> main by @bact in huggingface/tokenizers#1292

import Tuple from typing by @kellymarchisio in huggingface/tokenizers#1295

Fixing clippy warnings on 1.71. by @Narsil in huggingface/tokenizers#1296

Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node by @dependabot in huggingface/tokenizers#1299

feat: Added CITATION.cff. by @SamuelLarkin in huggingface/tokenizers#1302

Single warning for holes. by @Narsil in huggingface/tokenizers#1303

Give error when initializing tokenizer with too high stride by @boyleconnor in huggingface/tokenizers#1306

Handle when precompiled charsmap is empty by @kellymarchisio in huggingface/tokenizers#1308

Derive clone for TrainerWrapper by @jonatanklosko in huggingface/tokenizers#1317

CD backports by @chris-ha458 in huggingface/tokenizers#1318

0.13.4.rc1 by @Narsil in huggingface/tokenizers#1319

Release all at once for simplicity. by @Narsil in huggingface/tokenizers#1320

Fix stride condition. by @Narsil in huggingface/tokenizers#1321

pyo3: update to 0.19 by @mikelui in huggingface/tokenizers#1322

Add expect() for disabling truncation by @boyleconnor in huggingface/tokenizers#1316

Re-using scritpts from safetensors. by @Narsil in huggingface/tokenizers#1328

Reduce number of different revisions by 1 by @Narsil in huggingface/tokenizers#1329

Python 38 arm by @Narsil in huggingface/tokenizers#1330

Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in huggingface/tokenizers#1331

Updating the docs with the new command. by @Narsil in huggingface/tokenizers#1333

Update added tokens by @ArthurZucker in huggingface/tokenizers#1335

... (truncated)

Commits

9a79cb8 update Cargo.Toml
61a28c1 update
499bbb5 update rust release
5ff86dc add cargo locks
ccc8da4 fix unexpected . after patch number
673d026 update depencency versions
3dce63f Merge pull request #1335 from ArthurZucker/update-added-tokens
efec086 get_added_tokens_decoder returns BTREEMap
a7ace44 python stub.py
f435af8 linting
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

LoicGrobol / zeldarose

Update tokenizers requirement from <0.14,>=0.10 to >=0.10,<0.15 #64

v0.14.0

What's Changed