Fix/sp chars spm - Githubissues

mead-ml / mead-baseline

Deep-Learning Model Exploration and Development for NLP

Apache License 2.0

243 stars 73 forks source link

Fix/sp chars spm #908

Closed dpressel closed 2 years ago

dpressel commented 2 years ago

Change the SPM tokenization to support GPT2 pre-tok
Add Pile processing support
Fix preproc_tlm to not truncate ever