We've been working on a re-implementation of the text tokenizer in Rust, with bindings for Python, called instant-clip-tokenizer.
In our benchmarks it is around 70x faster than the current Python implementation.
Are you interested in switching to this library instead of the tokenizer included in repository? If yes I'm happy to send in a PR!
We've been working on a re-implementation of the text tokenizer in Rust, with bindings for Python, called instant-clip-tokenizer. In our benchmarks it is around 70x faster than the current Python implementation.
Are you interested in switching to this library instead of the tokenizer included in repository? If yes I'm happy to send in a PR!