Open flowpoint opened 3 years ago
Thanks for this nice code.
Transformers >3 uses fast rust tokenizers by default, so i had to add use_fast=False. Also some options were renamed. see here for more information: https://github.com/huggingface/transformers/releases?after=v4.1.1
A .gitignore was added so it wouldn't commit data and models to the repo.
I tested the scripts with the commands in the readmes, but a small epoch and batch size on torch1.8.1+rocm4.0.1 and transformers 4.8.1
Thanks for this nice code.
Transformers >3 uses fast rust tokenizers by default, so i had to add use_fast=False. Also some options were renamed. see here for more information: https://github.com/huggingface/transformers/releases?after=v4.1.1
A .gitignore was added so it wouldn't commit data and models to the repo.
I tested the scripts with the commands in the readmes, but a small epoch and batch size on torch1.8.1+rocm4.0.1 and transformers 4.8.1