JohannesGaessler
commented
6 days ago It is basically impossible to change WARP_SIZE from 32 for the CUDA code. The code is very tightly coupled to that specific value, not only for performance but also for shared memory allocation and correctness (because the code assumes that all threads in a warp are automatically synchronized). And to my knowledge there isn't a single NVIDIA GPU with a different warp size anyways. So this would at most provide benefit for ROCm/MUSA where a translation of the CUDA code is suboptimal in the first place. Writing dedicated code for other GPU architectures would both be less work and result in better performance than trying to change WARP_SIZE.
Looking through the code it appears that warp sizes are often hardcoded at 32 with some places supporting both 16 and 32. While the vulkan backend recognizes warp sizes of 64 correctly, and uses about 50% more power for it, performance is only marginally better, if not equal to, the CUDA/ROCM implementation using a warp size of 32.
It seems there is a big opportunity for improvement here, but the fact that F16 is supported with CUDA/HIP and not by vulkan complicates things.
I lack experience with optimizing for GPUs so I figured I would see if anyone has looked into doing this before. How involved would this change be?