Add Left Padding and Padding to Max Length

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension

Apache License 2.0

17 stars 13 forks source link

Closed apaniukov closed 1 month ago

apaniukov commented 1 month ago

CVS-131898 CVS-132172

Support left padding
Support padding to max_length
Change max_length during CLI conversion
Detokenizer skip special tokens by default
Update tests pass-rate calculation - exclude skipped tests
HF tokenizer object attributes for padding and truncation now have priority over values from tokenizer.json for better alignment

ilya-lavrenov commented 1 month ago

can we add tests?

apaniukov commented 1 month ago

can we add tests?

Working on it

ilya-lavrenov commented 1 month ago

Looks like current OV tokenizers master is broken against latest OV master. Two other PRs to OV tokenizers master also shows access violation issue.