Support for HF Tokenizers

guillaume-be / rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

https://docs.rs/crate/rust-bert

Apache License 2.0

2.67k stars 216 forks source link

Closed guillaume-be closed 1 year ago

guillaume-be commented 1 year ago

Add experimental support for HF tokenizers
Feature must be enabled with hf-tokenizers, created via TokenizerOption::from_hf_tokenizer_file(tokenizer_file, special_token_map_file)