Closed timmoon10 closed 1 month ago
The process group for FP8 amax reductions should be the tensor-parallel group, but we use the data-parallel group in the docs for multi-GPU training. See https://github.com/NVIDIA/TransformerEngine/issues/656#issuecomment-1972000820.
Fixes https://github.com/NVIDIA/TransformerEngine/issues/656.
Description
The process group for FP8 amax reductions should be the tensor-parallel group, but we use the data-parallel group in the docs for multi-GPU training. See https://github.com/NVIDIA/TransformerEngine/issues/656#issuecomment-1972000820.
Fixes https://github.com/NVIDIA/TransformerEngine/issues/656.
Type of change
Changes
Checklist: