Hello, can Megatron-DeepSpeed pre-train llama2?

bigscience-workshop / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Other

1.3k stars 211 forks source link

Open 13416157913 opened 10 months ago

13416157913 commented 10 months ago

Hello, can Megatron-DeepSpeed pre-train llama2? Can give a sample script?