search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.92k stars 776 forks source link

Make `onig` crate non-optional #1533

Closed nathaniel-daniel closed 2 months ago

nathaniel-daniel commented 4 months ago

This PR allows the Rust crate to be used without default features.

I ran into trouble trying to add this crate to a project without its default features. Looks like the onig crate has become a non-optional dependency of this crate as this crate will not compile without it. To reflect this, I made onig a required dependency and added a placeholder feature for backwards compatibility.

If adding multiple regex backends like #1510 is desired, this PR should be reworked to detect multiple enabled regex backends and no enabled regex backends and throw a compile error.

HuggingFaceDocBuilderDev commented 3 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.