Cuda support not working on WSL2

[Juff-Ma]

Hello, i tried testing an app with CUDA on WSL2, on Windows it worked perfectly fine but in WSL it doesn't detect a Cuda device but only the CPU accelerator. The nvidia deviceQuery sample and nvidia-smi detect my GPU fine and can run CUDA on it.

here is my test code:

using ILGPU;
using ILGPU.Runtime;

const float PI = (float)Math.PI;

using Context context = Context.Create(builder => builder.AllAccelerators());

Device device = context.Devices.First(x => x.AcceleratorType == AcceleratorType.Cuda);

using Accelerator accelerator = device.CreateAccelerator(context);

var deviceData = accelerator.Allocate1D(Enumerable
    .Repeat(0, 10)
    .Select(_ => Random.Shared.Next(0, 500))
    .ToArray());
var deviceOutput = accelerator.Allocate1D<float>(100_000);

var loadedKernel = accelerator.LoadAutoGroupedStreamKernel(
(Index1D i, ArrayView<int> data, ArrayView<float> output) =>
    output[i] = data[i % data.Length] * PI
);

loadedKernel((int)deviceOutput.Length, deviceData.View, deviceOutput.View);

accelerator.Synchronize();

foreach (float i in deviceOutput.GetAsArray1D())
{
    Console.WriteLine(i);
}

[Juff-Ma]

Addition, when using OpenCL in WSL2 (intel iGPU) it works completely fine

[MoFtZ]

welcome @Juff-Ma. I'm not sure anyone has previously tried ILGPU on WSL2, however, I'm surprised that the Nvidia sample projects work and ILGPU is not able to find the GPU.

ILGPU will attempt to find the Cuda drivers. On Linux, this should be libcuda.so. It could be that WSL2 puts the drivers in a different location. Or that ILGPU found the drivers, but for some other reason, determined that it was not compatible.

Will investigate when I have a chance.

[Yey007]

ILGPU will attempt to find the Cuda drivers. On Linux, this should be libcuda.so. It could be that WSL2 puts the drivers in a different location.

@MoFtZ I believe this is the case. In WSL libcuda.so seems to be under a special wsl directory (/usr/lib/wsl/lib). Would this prevent ILGPU from finding the driver? Do we need to make ILGPU search this path as well?

Edit: I can verify that the CUDA accelerator is not found on my WSL install either.

[Juff-Ma]

Yes,not only the cuda library is under /usr/lib/wsl/lib but CUDA itself is not in path (the installer didn't put /usr/local/cuda/bin in path and /usr/local/cuda/lib64 in the linker path)

[Yey007]

Sorry for the long wait but I think I'll try to take a look at fixing this today.

[Yey007]

Upon further research, this seems to be an issue with WSL rather than with ILGPU. I don't exactly understand what causes it, but the dynamic linker just cannot find libcuda.so on WSL for some reason, though libcuda.so.1 works fine.

I think we have two options at this point:

1. Wait for Microsoft to resolve the issue and, in the meantime, use this pretty simple workaround.
2. Search for libcuda.so.1 instead of libcuda.so when trying to load the library. This is as simple as changing a string, and I have implemented it here. I'm not sure if this works on regular Linux so testing would be appreciated, but it works fine on WSL. I'll hopefully get to testing on regular Linux soon as well.

Any thoughts on which approach we should move forward with?