Closed boss-chanon closed 1 year ago
Make spm_trainer can switch between spm and bpe mode for train sentencepiece tokenizer
spm_trainer
eos_token_id
0
Close #
Why this PR
Make
spm_trainer
can switch between spm and bpe mode for train sentencepiece tokenizerChanges
spm_trainer
can select mode for traineos_token_id
separator sentence when tokenize sentence instead0
Related Issues
Close #
Checklist