Bump tokenizers from 0.15.2 to 0.19.1

[dependabot[bot]]

Bumps tokenizers from 0.15.2 to 0.19.1.

Release notes

Sourced from tokenizers's releases.

v0.19.1

What's Changed

• add serialization for ignore_merges by @​ArthurZucker in huggingface/tokenizers#1504

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1

v0.19.0

What's Changed

• chore: Remove CLI - this was originally intended for local development by @​bryantbiggs in huggingface/tokenizers#1442
• [remove black] And use ruff by @​ArthurZucker in huggingface/tokenizers#1436
• Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @​dependabot in huggingface/tokenizers#1456
• Added ability to inspect a 'Sequence' decoder and the AddedVocabulary. by @​eaplatanios in huggingface/tokenizers#1443
• 🚨🚨 BREAKING CHANGE 🚨🚨: (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @​ArthurZucker in huggingface/tokenizers#1476
• Add more support for tiktoken based tokenizers by @​ArthurZucker in huggingface/tokenizers#1493
• PyO3 0.21. by @​Narsil in huggingface/tokenizers#1494
• Remove 3.13 (potential undefined behavior.) by @​Narsil in huggingface/tokenizers#1497
• Bumping all versions 3 times (ty transformers :) ) by @​Narsil in huggingface/tokenizers#1498
• Fixing doc. by @​Narsil in huggingface/tokenizers#1499

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.2...v0.19.0

v0.19.0rc0

Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177

What's Changed

• chore: Remove CLI - this was originally intended for local development by @​bryantbiggs in huggingface/tokenizers#1442
• [remove black] And use ruff by @​ArthurZucker in huggingface/tokenizers#1436
• Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @​dependabot in huggingface/tokenizers#1456
• Added ability to inspect a 'Sequence' decoder and the AddedVocabulary. by @​eaplatanios in huggingface/tokenizers#1443
• 🚨🚨 BREAKING CHANGE 🚨🚨: (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @​ArthurZucker in huggingface/tokenizers#1476
• Add more support for tiktoken based tokenizers by @​ArthurZucker in huggingface/tokenizers#1493
• PyO3 0.21. by @​Narsil in huggingface/tokenizers#1494
• Remove 3.13 (potential undefined behavior.) by @​Narsil in huggingface/tokenizers#1497
• Bumping all versions 3 times (ty transformers :) ) by @​Narsil in huggingface/tokenizers#1498

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.2...v0.19.0rc0

Commits

• 3b3c960 version 0.19.1
• da30768 "0.19.1-rc0" and Cargo.lock
• 7733bc2 add serialization for ignore_merges (#1504)
• 91393ef Fixing doc. (#1499)
• 949d9e3 Bumping all versions 3 times (ty transformers :) ) (#1498)
• e0defa7 Remove 3.13 (potential undefined behavior.) (#1497)
• d5a8cc7 PyO3 0.21. (#1494)
• 914576f Add more support for tiktoken based tokenizers (#1493)
• 6e58f83 version = "0.16.0-dev.0"
• 0906971 Refactor metaspace (#1476)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[viking-sudo-rm]

@dependabot rebase