This PR reorganizes the tokenizers into formerbox/data/tokenizers module (thus reducing the risk of getting into circular imports issue). We also introduced GPT2 and BART tokenizers (alongside with present RoBERTa tokenizer) with associated tokenizer trainers.
Summary
This PR reorganizes the tokenizers into
formerbox/data/tokenizersmodule (thus reducing the risk of getting into circular imports issue). We also introduced GPT2 and BART tokenizers (alongside with present RoBERTa tokenizer) with associated tokenizer trainers.