Closed gwenzek closed 8 months ago
Hi @gwenzek, calling compute() on its own does not take into account the number of processes. If not using the toolkit's sync_and_compute, you'd need to manually multiply by the number of processes running.
e.g. in the code snippet you have above:
else:
val = metrics["tps"].compute().item() * get_world_size() # scale by number of processes running
You can see the example here for calculating the estimated vs exact global throughput values: https://github.com/pytorch/torcheval/blob/2261be49267f19bf2ec98d0b7f78895adcf9d4ac/examples/distributed_example.py#L146-L158
The assumption the estimated throughput makes is that all ranks are roughly equal in terms of time spent. However, the slowest rank dictates the overall performance in fully synchronous training, which is why the throughput metric takes the max across elapsed time counts when syncing states:
closing since it seems this was not a bug
🐛 Describe the bug
I'm using
torcheval.metrics.Throughput
to compute a number of "tokens per second" for my training loop. To avoid synchronization overhead I don't sync the metric every time I log it, but only once in a while. In this examples I'm using a 8 process job with 8 GPUsThis creates big spikes in the log graph, because throughput is not averaging over the number of workers, so when I sync I get x8 TPS.
Is this the expected behavior ? I would find it less surprising if sync was returning the average throughput, or if non-sync was returning an estimated global throughput.
Versions