search

lucidrains / deep-daze

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun
MIT License
4.37k stars 327 forks source link

Nvidia Quadro RTX 5000 GPU utilization is 1-2% only. How to increase GPU utilization, thus the speed of the application? #117

Open ahmedahmedov opened 3 years ago

ahmedahmedov commented 3 years ago

Capture2

atheleon commented 3 years ago

I have this same issue- the program appears to primarily use CPU and very little GPU.

paulrobello commented 3 years ago

Same here NVIDIA RTX 2800

Alec55555 commented 3 years ago

You have to switch a window, for example the "copy" window into "Cuda", then it will look as you would've expected it.

ahmedahmedov commented 3 years ago

@Alec55555 I am sorry, I didn't really understand what you mean by "copy window into Cuda". I am simply following the tutorial as it is. How do I do what you have suggested?

Alec55555 commented 3 years ago

@ahmedahmedov No problem, use this tutorial https://michaelceber.medium.com/gpu-monitoring-on-windows-10-for-machine-learning-cuda-41088de86d65

soberirving commented 3 years ago

@Alec55555 After switching the window, it stll shows that GPU utilization is 0%, although the reply of torch.cuda.is_available() is TRUE...Can you give me your conda list? I want to know which part do I miss...Thank you very much!

TaylorBurnham commented 3 years ago

@ahmedahmedov and @soberirving this article can give you some more detail on what the entries in Task Manager mean. Since you are not 3D rendering you will not see much show up under 3D. Like what @Alec55555 said you have to switch the graph to the CUDA view.

image

This is what you are expecting to see for your GPU under CUDA.

image

You can also run nvidia-smi to get some more detail but it will show limited information. This is because on Windows the Driver Display Manager (WDDM) manages memory.

$ ./nvidia-smi.exe
Fri May  7 09:28:29 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.89       Driver Version: 465.89       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:09:00.0 Off |                  N/A |
| 58%   68C    P2   171W / 175W |   6743MiB /  8192MiB |     83%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        32    C+G   ...8bbwe\WindowsTerminal.exe    N/A      |
|    0   N/A  N/A      1356    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      1364    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      2008    C+G   Insufficient Permissions        N/A      |
+-----------------------------------------------------------------------------+

Compared to a Linux system you will have far more information available to you.

Fri May  7 13:24:39 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   41C    P0    71W /  70W |  12538MiB / 15109MiB |     82%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      4937      C   ...untu/deep/env/bin/python3    12535MiB |
+-----------------------------------------------------------------------------+