Sentencepiece Left Padding - Githubissues

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension

Apache License 2.0

17 stars 13 forks source link

Sentencepiece Left Padding #186

Closed apaniukov closed 2 weeks ago

apaniukov commented 3 weeks ago

Allow Batch Encoding For ChatGLM
Add Left Padding Option To Sentencepiece
Add per-test diff for better CI debugging
Turn off NPU building in CI