Open WeihongM opened 2 years ago
Thank you for your interest in our work! We have no plans to provide a pre-training code yet. You can implement your own pre-training script by referring the Hugging Face's transformers code: https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/data/data_collator.py#L748
Thanks for your impressive work. Can you share how to implement pretraining code?