search

clEsperanto / pyclesperanto_prototype

GPU-accelerated bio-image analysis focusing on 3D+t microscopy image data
http://clesperanto.net
BSD 3-Clause "New" or "Revised" License
208 stars 44 forks source link

Jupyter notebook Kernel crashes while using pyclesperanto in Ubuntu 22.04.2 on WSL2, Windows 11 #317

Closed somas193 closed 1 year ago

somas193 commented 1 year ago

pyclesperanto_error I am trying to run the workflow: https://github.com/clEsperanto/pyclesperanto_prototype/blob/master/demo/tribolium_morphometry/tribolium_morphometry2.ipynb through VS Code on a system with Ubuntu 22.04.2 (Kernel:5.15.90.1-microsoft-standard-WSL2) installed in WSL2 on Windows 11. Initially, I wasn't able to select the GPU even after installing ocl-icd-system. Then, I also installed pocl and this made it possible to select between CPU and GPU. Now I am able to select the device but the Kernel crashes when I try to use any method from pyclesperanto. The same workflow runs without any problem on the host Windows system. The details are given below:

pyclesperanto version: 0.24.1 ocl-icd version: 2.3.1 ocl-icd-system version: 1.0.0 pocl version: 4.0 CPU: AMD Ryzen 9 5900X GPU: NVIDIA RTX A6000 nvidia-smi version: 510.47.03, driver version: 511.65

Any suggestions on how I can solve this issue?

thawn commented 1 year ago

iirc, pocl just emulates a GPU. In order to get access to the actual GPU from within WSL2, you need to install special NVIDIA drivers. Maybe this link helps (please report back if it does ;-) ): https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl

somas193 commented 1 year ago

I was under the assumption that the GPU setup works fine because I was able to query nvidia-smi on WSL and the workflow ran on the GPU on the host system. But I suppose that's not the case! I followed the instructions provided on the link above and I'm able to run the workflow on WSL now. For future reference, I did the following:

  1. Download appropriate drivers from: https://www.nvidia.com/Download/index.aspx?lang=en-us and install them on the host system (Windows 11).
  2. Install the appropriate driver on WSL from: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local through commands given on the website.
  3. Run wsl.exe -d Ubuntu --shutdown + wsl.exe --terminate Ubuntu (replace Ubuntu with the name of the distro installed in WSL) on the host system with powershell and restart WSL. The condition of the drivers can be checked by querying nvidia-smi and it should work.

Thanks for the help @thawn

somas193 commented 1 year ago
pyclesperanto_windows11 pyclesperanto_pocl

@thawn @haesleinhuepf while running the workflow: https://github.com/clEsperanto/pyclesperanto_prototype/blob/master/demo/tribolium_morphometry/tribolium_morphometry2.ipynb on WSL2, clesperanto selects the GPU runtime through the pocl-cuda driver and not the installed CUDA driver whereas on the host system i.e. Windows 11 it detects the installed CUDA driver. Is this defined behavior for clesperanto on Linux/WSL2? AFAIK pocl is supposed to be used for CPU right?

somas193 commented 1 year ago

On further digging, I came to know that OpenCL support through CUDA driver in WSL2 is actually an issue and has been listed on the official WSL repository: https://github.com/microsoft/WSL/issues/6951 So for now, pocl-cuda seems to be the only option on WSL2!

thawn commented 1 year ago

on WSL2, clesperanto selects the GPU runtime through the pocl-cuda driver and not the installed CUDA driver

afaik, pocl-cuda is supposed to use the GPU, although the performance may vary. Could you please check with nvidia-smi what the GPU load is, while you are running a pyclesperanto workflow? That way, you could tell if it is using the GPU or not.