search

Anush008 / fastembed-rs

Library for generating vector embeddings, reranking in Rust
https://docs.rs/fastembed
Apache License 2.0
264 stars 36 forks source link

Made tokenizer public #38

Closed prattcmp closed 4 months ago

prattcmp commented 7 months ago

The tokenizer should be public so it can be accessed for things like calculating token count.

Anush008 commented 4 months ago

Closing since done in https://github.com/Anush008/fastembed-rs/commit/df9fa57e5d046cba776c3467d5265e3e55c70dba.