Support operating computer system

[Southpika]

Hello, I want to use this package in my company, so I want to know the supported computer system when using pkg tokenizers, does it have a lowest version requirements in Linux/Mac/Windows? Besides this, I also want to know if I run it in C++ program, what should I pay attention to? Wish your answer, Many thanks.

[Narsil]

Pretty much everywhere on consumer hardware.

For using in C++ you probably want to link directly to the rust side using a C compatibility layer, which we do not provide. (it should be mostly creating a few functions calls: https://docs.rust-embedded.org/book/interoperability/c-with-rust.html

[Narsil]

Please not that tokenizers already uses threads, so do not use threading in your own library, or disable it it using TOKENIZERS_PARALLELISM=0 as an environment variable.