Open xrsrke opened 10 months ago
There have been new tensor parallelism techniques that improve over Megatron's tensor parallelism. We consider implementing either 2D, 2.5D or 3D tensor parallelism.
Reading
There have been new tensor parallelism techniques that improve over Megatron's tensor parallelism. We consider implementing either 2D, 2.5D or 3D tensor parallelism.
Reading