search

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension
Apache License 2.0
17 stars 13 forks source link

Added Tokens Support for BPE Tokenizers #144

Closed apaniukov closed 2 months ago

apaniukov commented 2 months ago

CVS-125252