how to pretrain t5-lm adapted?

bigscience-workshop / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Other

1.3k stars 211 forks source link

Open nanyyyyyy opened 1 year ago

nanyyyyyy commented 1 year ago

I saw some closed discussions about implementing t5-lm. But i didn’t see any script to initialize the pretrain. Thanks