search

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension
Apache License 2.0
17 stars 13 forks source link

Revert "Added Tokens Support for BPE Tokenizers" #146

Closed rkazants closed 2 months ago

rkazants commented 2 months ago

Reverts openvinotoolkit/openvino_tokenizers#144