BPE Fallback for Sentencepiece

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension

Apache License 2.0

17 stars 13 forks source link

Closed apaniukov closed 3 weeks ago

apaniukov commented 4 weeks ago

CVS-144675

Fallback to BPE TokenizerFast conversion if Sentencepiece tokenizer doesn't have .model file
Add Prepend normalizer support
Add ByteFallback decoding operation
Support Sequence decoding HF pipeline parsing