how many gpus do i need

microsoft / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Other

1.85k stars 342 forks source link

Open lyzKF opened 1 year ago

lyzKF commented 1 year ago

To train a model(7B) with megatron-deepspeed, tensor_parallelism=2 pipeline_parallelism=8

how many GPUS do i need?

flyingdown commented 1 year ago

Roughly calculated, about 280G of memory is required, which is divided into 16 cards, and each card occupies about 17G.