Support for warp sizes of 64

[Haus1]

Looking through the code it appears that warp sizes are often hardcoded at 32 with some places supporting both 16 and 32. While the vulkan backend recognizes warp sizes of 64 correctly, and uses about 50% more power for it, performance is only marginally better, if not equal to, the CUDA/ROCM implementation using a warp size of 32.

It seems there is a big opportunity for improvement here, but the fact that F16 is supported with CUDA/HIP and not by vulkan complicates things.

I lack experience with optimizing for GPUs so I figured I would see if anyone has looked into doing this before. How involved would this change be?

[JohannesGaessler]

It is basically impossible to change WARP_SIZE from 32 for the CUDA code. The code is very tightly coupled to that specific value, not only for performance but also for shared memory allocation and correctness (because the code assumes that all threads in a warp are automatically synchronized). And to my knowledge there isn't a single NVIDIA GPU with a different warp size anyways. So this would at most provide benefit for ROCm/MUSA where a translation of the CUDA code is suboptimal in the first place. Writing dedicated code for other GPU architectures would both be less work and result in better performance than trying to change WARP_SIZE.

[Haus1]

Yeah, it seems the hardware is designed to have 64 warps resident but limited register space only allows them to track 32 threads at a time. The rest seem to be used as a cache that can be swapped in as needed. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#features-and-technical-specifications-technical-specifications-per-compute-capability

Thanks for the insight