Closed ofey404 closed 1 year ago
If proper, I could make similar change to remaining tensor_parallel_*.py
.
Hi @ofey404 thank you for your contribution! @kurisusnowdeng Could you please help review this PR? Thanks.
Hi, you don't need to do all-reduce in your cutomized models, as all-reduce is done in col_nn.Linear
.
See https://github.com/hpcaitech/ColossalAI/blob/91a5999825137ffb4d575b21bf4c6cb41033161a/colossalai/nn/layer/parallel_1d/layers.py#L664
Tutorial 1D Tensor Parallelism mentioned the use of
all_reduce()
, but the example attached doesn't show us how to do it.Quote:
So I made this enhancement, to print weight information before and after calling
all_reduce()
.Output: