[Common] Added JIT-compiled fused cast transpose kernels

[Oleg-Goncharov]

Description

This PR extends the JIT-compilation on the fused cast transpose kernels using CUDA Driver API (nvrtc), which enables performance optimizations otherwise not available during runtime, and reduces the size of the TE binary libtransformer_engine.so by 4.5% (from 292M to 279M). The performance was benchmarked for the cast_transpose_dbias and cast_transpose_dbias_dgelu kernels for the matrix size 2048x12288 and different combinations of input types. The results are provided in the table below:

NOTES:

• Original - fused cast transpose kernels before the commit #884
• Template - current state
• JIT - proposed by this PR
• Benchmarked on H100 HBM3
• Time measured in microseconds

Performance of the kernels remains the same, with some marginal changes (~2-6%) of nvte functions runtime.

Type of change

• [ ] Documentation change (change only to the documentation, either a fix or a new content)
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Changes

• Modified the logic in the cast_transpose_fusion.cu
• Added the corresponding source file into the transformer_engine/common/transpose/rtc folder
• Adjusted the CMake for generating a string header from that file

Checklist:

• [x] I have read and followed the contributing guidelines
• [x] The functionality is complete
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes

[Oleg-Goncharov]

/te-ci

[Oleg-Goncharov]

/te-ci

[Oleg-Goncharov]

/te-ci

[Oleg-Goncharov]

/te-ci

[Oleg-Goncharov]

/te-ci

[Oleg-Goncharov]

/te-ci