search

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension
Apache License 2.0
18 stars 16 forks source link

Add Sequence Post-Processor Support #125

Closed apaniukov closed 4 months ago

apaniukov commented 4 months ago
mryzhov commented 4 months ago

do we need to merge it to release branches too?

apaniukov commented 4 months ago

do we need to merge it to release branches too?

We could port the first commit with the fix for llama3 tokenizer to 2024.1.0.1 release.