Closed SaltyChiang closed 5 months ago
I think I've hit this in the past (in WSL) and lost track of actually reporting it, I believe it's downstream of some constraints with WSL: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications
If it's easy for you (I've since broken my WSL setup...), can you modify lib/targets/cuda/device.cpp
around line 97 and have it print the value of the field deviceProp.hostNativeAtomicSupported
? My hunch is that will return 0
in WSL (and 1
in proper Linux environments). If that's the case, we should add a runtime check for QUDA_HETEROGENEOUS_ATOMICS && deveProp.hostNativeAtomicSupported == 1
.
Thank you for your reply! I didn't realize that the atomic feature is related to UM.
You are right, deviceProp.hostNativeAtomicSupported
is 0
on my machine. But it's still strange that Tesla P100 works fine on WSL2, and I will check the property on P100 later.
Oh, that's strange, I misunderstood your original post, I didn't realize it was working in WSL2 with the Tesla P100... so I'm a bit confused there. Ah well! I'll still be interested in seeing what deviceProp.hostNativeAtomicSupported
returns on your WSL2+Tesla P100 configuration. If it returns 1
there (for whatever reason I can't necessarily conceive of), that means we at least still have a robust runtime check, even if the reason behind it working on Tesla vs GeForce is a bit of a mystery.
The problem is more complicated than I thought. I tried to call qudaDeviceSynchronize
after TunableKernel::launch_device<Functor, grid_stride>(KERNEL(Reduction2D), tp, stream, arg)
in include/tunable_reduction.h
and the test passed with good performance comparing with disabling heterogeneous atomic. It seems that the while
sentence in the original post happens too early.
deviceProp.hostNativeAtomicSupported
returns 0
on Tesla P100, so this issue might not be due to the atomic feature.
Closing this out. For a variety of reasons, Windows does not support cuda::std::atomic
when running on Pascal architecture, and Volta upwards is required. As @SaltyChiang has found, when running with Pascal on Windows, one needs to set QUDA_HETEROGENEOUS_ATOMICS=OFF
when compiling QUDA.
I want to run QUDA on my GeForce RTX 3080 12GB, and I have successfully built
invert_test
with the latest commit in Debian testing (WSL2, which is actually a virtual machine on Windows). The test seems to block after printingcublasCreated successfully
.I found that the
HETEROGENEOUS_ATOMIC
macro enables subroutines ininclude/targets/cuda/reduce_helper
and thewhile
sentence in line 131 never finishes. I tried to set-DQUDA_HETEROGENEOUS_ATOMIC=OFF
and then the test gave me a good performance.This should not be a Windows' or WSL2's problem, because the test works pretty fine on Tesla P100 while using the same environment.
Is this a limitation of GeForce GPUs? Or I just did something wrong?