issues
search
openvinotoolkit
/
openvino_tokenizers
OpenVINO Tokenizers extension
Apache License 2.0
18
stars
16
forks
source link
Llama3 Tokenizer Support
#133
Closed
apaniukov
closed
3 months ago
apaniukov
commented
3 months ago
Cherry-pick from master:
Add support for Sequence post-processor
Add not-add-special-tokens conversion flag
Cherry-pick from master: