The sentencepiece model cannot add bos_token when there is no bos_token in the dictionary. In such situations add_eos=True leads to a failed check inside the sentencepiece library. Modify the add_bos_token flag logic to avoid such situations.
There is a regression for camembert-base_slow tokenizer that is not caused by a bug fix. Had to lower the pass rate to not block the fix.
CVS-133826
The sentencepiece model cannot add bos_token when there is no bos_token in the dictionary. In such situations
add_eos=Trueleads to a failed check inside the sentencepiece library. Modify theadd_bos_tokenflag logic to avoid such situations.There is a regression for
camembert-base_slowtokenizer that is not caused by a bug fix. Had to lower the pass rate to not block the fix.