GuanhuaWang
commented
1 year ago Hi @stas00 , Thanks for raising this interesting question.
The tl;dr answer is, to get reasonable GPU throughput when training at scale (64+GPUs), 100 Gbps is not enough, 200-400 Gbps is ok, 800-1000 Gbps will be ideal.
Hardware cost-effectiveness: Given the price that InfiniBand (IB) is usually more expensive than ethernet, 200 to 400/800 Gbps ethernet link seems to be a more cost-effective solution. RDMA over Converged Ethernet (RoCE) could achieve similar throughput performance comparing with IB (but slight longer latency for small message passing).
Below is the math
Notation: model size M, num of GPU per node is G, num of Node is N, in total G*N GPUs in use
Assumption: intra-node GPU communication overhead is ignored (Since NVLink/NVSwitch are high-throughput links)
In ZeRO-3, we have
all-gatheron weights (M) in forward, thenall-gatheron weights (M) in backward, last isreduce-scatteron gradients (M) in backward. In total 3 global collective calls.For each of above 3 collectives, each GPU need to sent out M/(G*N) data outside the node as cross-node traffic. Each node need to send out M / N
Given that we usually use fp16 (2 bytes) to represent both weights and graidents, for each collective, each node send out 2M/N Bytes. 3 collectives in total needs each node to send out 6M/N Bytes, which is equal to 8 6 M/N = 48 M / N bits.
Numbers we collected over 384 V100 GPUs (24 DGX-2 nodes) and 176B model is,
- With 100 Gbps IB, we only have <20 TFLOPs per GPU.
- With 200-400 Gbps IB, we achieve reasonable TFLOPs around 30-40 per GPU.
- For 800 Gbps IB, we reach 40+ TFLOPs per GPU.
We also notice that when training at scale, the communication overhead is more pronounced with small micro-batch size per GPU. And we may not be able to increase micro-batch size since global-batch size is often fixed to achieve good model convergence rate. We are trying to solve this issue with our up-coming new release project called ZeRO++, which could achieve better e2e system throughput when training at scale with small micro-batch size using ZeRO-3. Stay tuned!
stas00
thomasw21
tjruwase
wptoux
deanpeterson
Given a model size and number of gpus, how can we calculate what kind of throughput should the interconnect network have to handle ZeRO-3 traffic. Is 100Gbps enough? or does one need 1_000Gbps?
Users need to know what kind of requirements they need to seek out of the setup they buy or rent in order not to be network-bound and overpay for idling gpus.
Of course, this would also require the knowledge of gpus since in order not to be network-bound we need to ensure that comms <= compute+memory movement. We are of course discussing the compute/comms overlap here.
But even w/o having the knowledge of compute timing, it should be straightforward to calculate that to train a model of size X on Y gpus, this much traffic will be incurred for zero shards and that much for various reductions.
For ZeRO inference the need would be just zero shards traffic.
Thank you.
Anecdotally, when we were choosing which framework to choose to train BLOOM-176 we had none of these numbers and had to benchmark the actual cluster, and measure the overall throughput., which for many users can be very difficult to procure before they commit to buying/renting hardware. It'd have helped a lot to know that for 176B parameter models using ZeRO3 on 384 gpus it'd require that many Gbps network.