search

microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.48k stars 822 forks source link

Running CUDA inside Docker\WSL on Windows 11 Hyper-v VM with GPU Patritioning #12097

Open PolyglotBuckeye opened 1 month ago

PolyglotBuckeye commented 1 month ago

Is your feature request related to a problem? Please describe.

The problem is that WSL under a Hyper-V GPU Partitioned machine can't use the GPU (Nvidia), as stated in #8179 2 YEARS ago.

Describe the solution you'd like

Expected behavior on vm: https://www.youtube.com/watch?v=JaHVsZa2jTc

For this to work there should be some configuration regarding the linkers to the drivers that are virtualized inside the Hyper-v VM.

The files from "C:\Windows\System32\lxss" when copied to the VM should do the link to the drivers. Or they should be changed in a way to accommodate the virtual GPU.

Describe alternatives you've considered

I'll try to do GPU partitioning with a Ubuntu VM under Hyper-v.

Additional context

Tech stack: Windows 11 ✅ → Hyper-v ✅-> Windows 11 guest + Gpu Partitioning + Nested Virtualization ✅-> Docker Desktop + WSL ✅ → GPU acceleration (nvidia-smi) ❌

This is what I get in WSL when I copy the linkers("C:\Windows\System32\lxss") to VM:

    Ubuntu:$ nvidia-smi

    Failed to initialize NVML: GPU access blocked by the operating system

    Failed to properly shut down NVML: GPU access blocked by the operating system

More details and my hardware are here: https://forums.developer.nvidia.com/t/run-cuda-inside-docker-wsl-on-windows-11-hyper-v-vm-with-gpu-patritioning/308212

Cuda #WSL #GPU-P #Paravirtualization #GPU Partitioning #NVIDIA #Docker #Nested Virtualization #nvidia-smi #Hyper-v

github-actions[bot] commented 1 month ago

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'. Otherwise please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs Download and execute [collect-wsl-logs.ps1](https://github.com/Microsoft/WSL/blob/master/diagnostics/collect-wsl-logs.ps1) in an **administrative powershell prompt**: ``` Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1 Set-ExecutionPolicy Bypass -Scope Process -Force .\collect-wsl-logs.ps1 ``` The script will output the path of the log file once done. If this is a networking issue, please use [collect-networking-logs.ps1](https://github.com/Microsoft/WSL/blob/master/diagnostics/collect-networking-logs.ps1), following the instructions [here](https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#collect-wsl-logs-for-networking-issues) Once completed please upload the output files to this Github issue. [Click here for more info on logging](https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#8-collect-wsl-logs-recommended-method) If you choose to email these logs instead of attaching to the bug, please send them to wsl-gh-logs@microsoft.com with the number of the github issue in the subject, and in the message a link to your comment in the github issue and reply with '/emailed-logs'.

View similar issues

Please view the issues below to see if they solve your problem, and if the issue describes your problem please consider closing this one and thumbs upping the other issue to help us prioritize it!

Open similar issues:

Note: You can give me feedback by thumbs upping or thumbs downing this comment.

PolyglotBuckeye commented 1 month ago

/feature

github-actions[bot] commented 1 month ago
Diagnostic information ``` Found '/feature', adding tag 'feature' ```
sirredbeard commented 2 weeks ago

I'll try to do GPU partitioning with a Ubuntu VM under Hyper-v.

I am curious what would happen in this scenario.

In my experience working with partitioned GPUs on Azure (but admittedly never on WSL) requires the NVIDIA GRID drivers.