Suggestions for implement pre-training

[WeihongM]

Thanks for your impressive work. Can you share how to implement pretraining code?

[tghong]

Thank you for your interest in our work! We have no plans to provide a pre-training code yet. You can implement your own pre-training script by referring the Hugging Face's transformers code: https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/data/data_collator.py#L748