search

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension
Apache License 2.0
18 stars 16 forks source link

Port qwen eos token to 2024.0 #114

Closed ilya-lavrenov closed 4 months ago

ilya-lavrenov commented 4 months ago

Required for GenAI repo, which uses latest released version. Ported https://github.com/openvinotoolkit/openvino_tokenizers/pull/109