Any possibility/plan to support fused kernels?

NVIDIA / nccl

Optimized primitives for collective multi-GPU communication

Other

3.27k stars 826 forks source link

Any possibility/plan to support fused kernels? #1522

Open dearsxx0918 opened 1 day ago

dearsxx0918 commented 1 day ago

Hi sjeaugey, Since many customers customize there allreuce+rms_norm, GEMM+reducescatter/allgather, do we have any plan to support those fusion APIs in the future?

Best regards, -Edda