issues
search
openvinotoolkit
/
openvino_tokenizers
OpenVINO Tokenizers extension
Apache License 2.0
18
stars
16
forks
source link
Add Tokenizer Benchmarks
#119
Closed
apaniukov
closed
4 months ago
apaniukov
commented
4 months ago
Add vLLM-based tokenizer benchmark
Add llama3 tokenizer to tests