Closed laladong closed 3 weeks ago
in 3.3.1 All-to-all based implementation. ' Once a GPU receives gradients from its predecessor, we dequantize it to recover full precision and conduct a local reduction. ' I would like to know what is the specific operation of reduction?
@laladong - this is an all reduce, what more/specifically were you looking for?
Closing as stale for now, please re-open if you have more questions.
in 3.3.1 All-to-all based implementation.
' Once a GPU receives gradients from its predecessor, we dequantize it to recover full precision and conduct a local reduction. ' I would like to know what is the specific operation of reduction?