search

microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
16.9k stars 797 forks source link

TensorFlow GPU performance on WSL2 lags behind TensorFlow GPU and CPU performance on native Windows environments #10787

Open TP066335 opened 7 months ago

TP066335 commented 7 months ago

Windows Version

Microsoft Windows [Version 10.0.22631.2715]

WSL Version

2.0.9.0

Are you using WSL 1 or WSL 2?

Kernel Version

Linux version 5.15.133.1-microsoft-standard-WSL2

Distro Version

Ubuntu 22.04.1 LTS

Other Software

I'm using Jupyter Notebook to train a CNN model.

Repro Steps

I follow this youtube video to set up TensorFlow GPU on WSL2. https://www.youtube.com/watch?v=OHCPGvf06EA

I also follow this youtube video to set up Tensorflow GPU on native Windows environments. https://www.youtube.com/watch?v=QUjtDIalh0k&t=280s

After conduct the comparison on the performance, I notice TensorFlow GPU performance on WSL2 lags behind TensorFlow GPU and CPU performance on native Windows environments.

Expected Behavior

Tensorflow usually runs faster with a GPU compared to running on native Windows environments.

Actual Behavior

I was hoping Tensorflow GPU on WSL2 would at least be faster than running Tensorflow on Windows with just the CPU, but unfortunately, the GPU performance on WSL2 turns out to be the slowest option.

Diagnostic Logs

Attached herewithe the Jupyter Notebooks for your reference. Jupyter Notebook.zip

Experiment Results:

  1. Tensorflow GPU on Native Windows: 00:16:43
  2. Tensorflow CPU on Native Windows: 00:24:38
  3. Tensorflow GPU on WSL2: 00:35:13

Conclusion: The GPU performance on WSL2 turns out to be the slowest option.

GPU and CUDA Details: GPU and CUDA Details on WSL2.zip

iourit commented 6 months ago

@TP066335 Please take ETW traces when running the scenario in different modes and share. To take a trace:

iourit commented 6 months ago

Hi @TP066335. Thanks for the ETL file. In general, ETLs can be compressed very well, so it is easier to share the files. For example, the current file is compressed to ~98MB zip.

iourit commented 6 months ago

Hi @TP066335. Please send an ETL for Tensorflow GPU on Native Windows. Regarding the WSL2 ETL. There are 2 adapters (NVIDIA and AMD). Is the intent to run Tensorflow on the NVIDIA adapter? I see that there are 2 processes, which use GPU in WSL2 - python and Xwayland. All work is done on the AMD adapter. If the intent is to use the NVIDIA GPU, try to disable the AMD adapter in Device manager.