Fix Decoding For Long Tokens

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension

Apache License 2.0

17 stars 13 forks source link

Closed apaniukov closed 3 weeks ago

apaniukov commented 3 weeks ago

CVS-145223 CVS-144252

Add buffer for long token decoding (previously decoding tokens longer than 100 symbols might cause a segfault)
Ignore OOV tokens during decoding