Add Truncation To Sentencepiece - Githubissues

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension

Apache License 2.0

20 stars 16 forks source link

Add Truncation To Sentencepiece #225

Closed apaniukov closed 3 weeks ago

apaniukov commented 3 weeks ago

Add Basic Truncation To Sentencepiece

Move checks to helpers functions
Add tests for chat templates
Filter skipped tests when pass-rate calculated
Not use regex to add prefix space in sentencpiece
Add diff to test results
Fix chatglm + re2 splitting
Fix Gemma SP Model

CVS-132366, CVS-150380