dpressel
commented
2 years ago - Adds SPM "normal" vectorizer (no additional outside token handling)
- Adds dict1d and label vectorizer for GPT2/RoBERTa that allows tagging tasks
- Change the sentencepiece dep
- Update a few defaults in the pretraining, include SPM everywhere