search

Ricks-Lab / gpu-utils

A set of utilities for monitoring and customizing GPU performance
GNU General Public License v3.0
142 stars 23 forks source link

Major Release Plans #88

Closed Ricks-Lab closed 4 years ago

Ricks-Lab commented 4 years ago

The latest development work on the utilities has been to add Nvidia read capabilities. Also, the utilities now leverage generic PCIe sensor reading to detect all GPUs in a system. As a result, it may be appropriate to change the name of the project for the next major release. I was considering a name that would be consistent with what could be used for the package on Debian and a package released on PyPI. I was considering a project name of rickslab-gpu-utils and executables with names like rlgpu-ls, rlgpu-mon, rlgpu-pac, and rlgpu-plot.

Let me know your thoughts or any other recomendations.

Ricks-Lab commented 4 years ago

Well, I just uninstalled and reinstalled and got the same version level.

Successfully installed rickslab-gpu-utils-3.4.11

Is that your latest?

All the apps run correctly without errors on my cards.

You need to download from GitHub in order to test the code I am working on. I won’t package it until it is verified.

Ricks-Lab commented 4 years ago

@csecht Your latest results looks like nvidia-smi isn't accessible. But before you fix that can you run again with the latest? I won't to make sure it is robust enough to deal with all items not being read.

Ricks-Lab commented 4 years ago

@csecht Your latest results looks like nvidia-smi isn't accessible. But before you fix that can you run again with the latest? I won't to make sure it is robust enough to deal with all items not being read.

Never mind... I read your post to quickly. Actually said it was found. I fixed a typo that should fix the error.

csecht commented 4 years ago

The typo edit didn't fix it:

 ./gpu-ls
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Traceback (most recent call last):
  File "./gpu-ls", line 150, in <module>
    main()
  File "./gpu-ls", line 98, in main
    gpu_list.set_gpu_list(clinfo_flag=True)
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1849, in set_gpu_list
    self[gpu_uuid].read_gpu_sensor_set_nv()
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1281, in read_gpu_sensor_set_nv
    if results['power.min.limit'] and re.fullmatch(PATTERNS['IS_FLOAT'], results['power.min_limit']):
KeyError: 'power.min.limit'
Ricks-Lab commented 4 years ago

Sorry, just pushed another version. Now it is fixed.

csecht commented 4 years ago

Not fixed yet...

~/Desktop/gpu-utils-master$ ./gpu-ls
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Traceback (most recent call last):
  File "./gpu-ls", line 150, in <module>
    main()
  File "./gpu-ls", line 98, in main
    gpu_list.set_gpu_list(clinfo_flag=True)
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1850, in set_gpu_list
    self[gpu_uuid].read_gpu_sensor_set_nv()
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1304, in read_gpu_sensor_set_nv
    if results[sn_k] and re.fullmatch(PATTERNS['IS_FLOAT']):
TypeError: fullmatch() missing 1 required positional argument: 'string'
Ricks-Lab commented 4 years ago

Surprised the IDE didn't catch that one. I have pushed a fix.

csecht commented 4 years ago

It's hanging on an bad nvidia-smi call

:~/Desktop/gpu-utils-master$ ./gpu-ls
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: INTEL: 1, NVIDIA: 1, AMD: 1
AMD: amdgpu version: 20.10-1048554
AMD: Wattman features enabled: 0xfffd7fff
Traceback (most recent call last):
  File "./gpu-ls", line 150, in <module>
    main()
  File "./gpu-ls", line 121, in main
    gpu_list.read_gpu_sensor_set(data_type=Gpu.GpuItem.SensorSet.All)
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 2093, in read_gpu_sensor_set
    gpu.read_gpu_sensor_set(data_type)
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1234, in read_gpu_sensor_set
    return self.read_gpu_sensor_set_nv(data_type)
  File "/home/craig/gpu-utils-master/GPUmodules/GPUmodule.py", line 1304, in read_gpu_sensor_set_nv
    if results[sn_k] and re.fullmatch(PATTERNS['IS_FLOAT'], results[sn_k]):
KeyError: 'temperature.memory'

When run from nvidia-smi:

:~$ /usr/bin/nvidia-smi -i 01:00.0 --query-gpu=temperature.memory --format=csv
Field "temperature.memory" is not a valid field to query.
Ricks-Lab commented 4 years ago

I have pushed an update. Can you also post a debug log for the next one?

csecht commented 4 years ago

It worked! (U still want the debug? lol)

~/Desktop/gpu-utils-master$ ./gpu-ls --debug
Ubuntu: Validated
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: INTEL: 1, NVIDIA: 1, AMD: 1
AMD: amdgpu version: 20.10-1048554
AMD: Wattman features enabled: 0xfffd7fff
3 total GPUs, 1 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor: INTEL
   Readable: False
   Writable: False
   Compute: False
   Device ID: {'device': '0x3e91', 'subsystem_device': '0x8694', 'subsystem_vendor': '0x1043', 'vendor': '0x8086'}
   Decoded Device ID: 8th Gen Core Processor Gaussian Mixture Model
   Card Model: Intel Corporation 8th Gen Core Processor Gaussian Mixture Model
   PCIe ID: 00:02.0
   Driver: i915
   GPU Type: Unsupported
   HWmon: None
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:02.0

Card Number: 1
   Vendor: NVIDIA
   Readable: True
   Writable: False
   Compute: True
   GPU UID: GPU-fcbaadc4-4040-c2e5-d5b6-52d1547bcc64
   GPU S/N: [Not Supported]
   Device ID: {'device': '0x1381', 'subsystem_device': '0x1073', 'subsystem_vendor': '0x10de', 'vendor': '0x10de'}
   Decoded Device ID: GM107 [GeForce GTX 750]
   Card Model: GeForce GTX 750
   Display Card Model: GeForce GTX 750
   Card Index: 0
   PCIe ID: 01:00.0
      Link Speed: GEN3
      Link Width: 8
   ##################################################
   Driver: 390.138
   vBIOS Version: 82.07.32.00.32
   Compute Platform: OpenCL 1.2 CUDA
   Compute Mode: Default
   GPU Type: Supported
   HWmon: None
   Card Path: /sys/class/drm/card1/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0
   ##################################################
   Current Power (W): 1.360
   Power Cap (W): 38.50
      Power Cap Range (W): [30.0, 38.5]
   Fan Target Speed (rpm): None
   Current Fan PWM (%): 40.000
   ##################################################
   Current GPU Loading (%): 0
   Current Memory Loading (%): 0
   Current VRAM Usage (%): 20.387
      Current VRAM Used (GB): 0.195
      Total VRAM (GB): 0.958
   Current  Temps (C): {'temperature.gpu': 29.0}
   Current Clk Frequencies (MHz): {'clocks.gr': 1019.0, 'clocks.mem': 2505.0, 'clocks.sm': 1019.0, 'clocks.video': 917.0}
   Maximum Clk Frequencies (MHz): {'clocks.max.gr': 1293.0, 'clocks.max.mem': 2505.0, 'clocks.max.sm': 1293.0}
   Current SCLK P-State: [0, '']
   Power Profile Mode: [Not Supported]

Card Number: 2
   Vendor: AMD
   Readable: True
   Writable: True
   Compute: False
   GPU UID: None
   Device ID: {'device': '0x731f', 'subsystem_device': '0xe411', 'subsystem_vendor': '0x1da2', 'vendor': '0x1002'}
   Decoded Device ID: Radeon RX 5600 XT
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev ca)
   Display Card Model: Radeon RX 5600 XT
   PCIe ID: 04:00.0
      Link Speed: 16 GT/s
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-5E4111U-X4G
   Compute Platform: None
   GPU Type: CurvePts
   HWmon: /sys/class/drm/card2/device/hwmon/hwmon3
   Card Path: /sys/class/drm/card2/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/0000:03:00.0/0000:04:00.0
   ##################################################
   Current Power (W): 7.000
   Power Cap (W): 160.000
      Power Cap Range (W): [0, 192]
   Fan Enable: 0
   Fan PWM Mode: [2, 'Dynamic']
   Fan Target Speed (rpm): 0
   Current Fan Speed (rpm): 0
   Current Fan PWM (%): 0
      Fan Speed Range (rpm): [0, 3200]
      Fan PWM Range (%): [0, 100]
   ##################################################
   Current GPU Loading (%): 0
   Current Memory Loading (%): 0
   Current GTT Memory Usage (%): 0.218
      Current GTT Memory Used (GB): 0.013
      Total GTT Memory (GB): 5.984
   Current VRAM Usage (%): 0.271
      Current VRAM Used (GB): 0.016
      Total VRAM (GB): 5.984
   Current  Temps (C): {'edge': 35.0, 'junction': 35.0, 'mem': 38.0}
   Critical Temps (C): {'edge': 118.0, 'junction': 99.0, 'mem': 99.0}
   Current Voltages (V): {'vddgfx': 775}
   Current Clk Frequencies (MHz): {'mclk': 100.0, 'sclk': 1780.0}
   Current SCLK P-State: [2, '1780Mhz']
      SCLK Range: ['800Mhz', '1820Mhz']
   Current MCLK P-State: [0, '100Mhz']
      MCLK Range: ['625Mhz', '930Mhz']
   Power Profile Mode: 5-COMPUTE
   Power DPM Force Performance Level: manual
Ricks-Lab commented 4 years ago

Cool! I found what I think is a redundant read and have removed it. Can you try it again?

csecht commented 4 years ago

Here's the --table:

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: INTEL: 1, NVIDIA: 1, AMD: 1
AMD: amdgpu version: 20.10-1048554
AMD: Wattman features enabled: 0xfffd7fff
3 total GPUs, 1 rw, 1 r-only, 0 w-only

Status of Readable GPUs:
┌─────────────┬────────────────┬────────────────┐
│Card #       │card1           │card2           │
├─────────────┼────────────────┼────────────────┤
│Model        │GeForce GTX 750 │Radeon RX 5600 X│
│GPU Load %   │100             │99              │
│Mem Load %   │36              │75              │
│VRAM Usage % │89.501          │11.969          │
│GTT Usage %  │None            │0.432           │
│Power (W)    │14.72           │114.0           │
│Power Cap (W)│38.50           │160.0           │
│Energy (kWh) │0.0             │0.0             │
│T (C)        │39.0            │53.0            │
│VddGFX (mV)  │0               │931             │
│Fan Spd (%)  │40.0            │34              │
│Sclk (MHz)   │1163            │1780            │
│Sclk Pstate  │0               │2               │
│Mclk (MHz)   │2505            │875             │
│Mclk Pstate  │0               │3               │
│Perf Mode    │[Not Supported] │5-COMPUTE       │
└─────────────┴────────────────┴────────────────┘

and the full list

:~/Desktop/gpu-utils-master$ ./gpu-ls
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: INTEL: 1, NVIDIA: 1, AMD: 1
AMD: amdgpu version: 20.10-1048554
AMD: Wattman features enabled: 0xfffd7fff
3 total GPUs, 1 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor: INTEL
   Readable: False
   Writable: False
   Compute: False
   Device ID: {'device': '0x3e91', 'subsystem_device': '0x8694', 'subsystem_vendor': '0x1043', 'vendor': '0x8086'}
   Decoded Device ID: 8th Gen Core Processor Gaussian Mixture Model
   Card Model: Intel Corporation 8th Gen Core Processor Gaussian Mixture Model
   PCIe ID: 00:02.0
   Driver: i915
   GPU Type: Unsupported
   HWmon: None
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:02.0

Card Number: 1
   Vendor: NVIDIA
   Readable: True
   Writable: False
   Compute: True
   GPU UID: GPU-fcbaadc4-4040-c2e5-d5b6-52d1547bcc64
   GPU S/N: [Not Supported]
   Device ID: {'device': '0x1381', 'subsystem_device': '0x1073', 'subsystem_vendor': '0x10de', 'vendor': '0x10de'}
   Decoded Device ID: GM107 [GeForce GTX 750]
   Card Model: GeForce GTX 750
   Display Card Model: GeForce GTX 750
   Card Index: 0
   PCIe ID: 01:00.0
      Link Speed: GEN3
      Link Width: 8
   ##################################################
   Driver: 390.138
   vBIOS Version: 82.07.32.00.32
   Compute Platform: OpenCL 1.2 CUDA
   Compute Mode: Default
   GPU Type: Supported
   HWmon: None
   Card Path: /sys/class/drm/card1/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0
   ##################################################
   Current Power (W): 14.860
   Power Cap (W): 38.50
      Power Cap Range (W): [30.0, 38.5]
   Fan Target Speed (rpm): None
   Current Fan PWM (%): 40.000
   ##################################################
   Current GPU Loading (%): 100
   Current Memory Loading (%): 36
   Current VRAM Usage (%): 89.093
      Current VRAM Used (GB): 0.854
      Total VRAM (GB): 0.958
   Current  Temps (C): {'temperature.gpu': 42.0, 'temperature.memory': None}
   Current Clk Frequencies (MHz): {'clocks.gr': 1163.0, 'clocks.mem': 2505.0, 'clocks.sm': 1163.0, 'clocks.video': 1046.0}
   Maximum Clk Frequencies (MHz): {'clocks.max.gr': 1293.0, 'clocks.max.mem': 2505.0, 'clocks.max.sm': 1293.0}
   Current SCLK P-State: [0, '']
   Power Profile Mode: [Not Supported]

Card Number: 2
   Vendor: AMD
   Readable: True
   Writable: True
   Compute: False
   GPU UID: None
   Device ID: {'device': '0x731f', 'subsystem_device': '0xe411', 'subsystem_vendor': '0x1da2', 'vendor': '0x1002'}
   Decoded Device ID: Radeon RX 5600 XT
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev ca)
   Display Card Model: Radeon RX 5600 XT
   PCIe ID: 04:00.0
      Link Speed: 16 GT/s
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-5E4111U-X4G
   Compute Platform: None
   GPU Type: CurvePts
   HWmon: /sys/class/drm/card2/device/hwmon/hwmon3
   Card Path: /sys/class/drm/card2/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/0000:03:00.0/0000:04:00.0
   ##################################################
   Current Power (W): 91.000
   Power Cap (W): 160.000
      Power Cap Range (W): [0, 192]
   Fan Enable: 0
   Fan PWM Mode: [2, 'Dynamic']
   Fan Target Speed (rpm): 1143
   Current Fan Speed (rpm): 1143
   Current Fan PWM (%): 34
      Fan Speed Range (rpm): [0, 3200]
      Fan PWM Range (%): [0, 100]
   ##################################################
   Current GPU Loading (%): 84
   Current Memory Loading (%): 58
   Current GTT Memory Usage (%): 0.432
      Current GTT Memory Used (GB): 0.026
      Total GTT Memory (GB): 5.984
   Current VRAM Usage (%): 11.969
      Current VRAM Used (GB): 0.716
      Total VRAM (GB): 5.984
   Current  Temps (C): {'edge': 54.0, 'junction': 61.0, 'mem': 72.0}
   Critical Temps (C): {'edge': 118.0, 'junction': 99.0, 'mem': 99.0}
   Current Voltages (V): {'vddgfx': 925}
   Current Clk Frequencies (MHz): {'mclk': 875.0, 'sclk': 1780.0}
   Current SCLK P-State: [2, '1780Mhz']
      SCLK Range: ['800Mhz', '1820Mhz']
   Current MCLK P-State: [3, '875Mhz']
      MCLK Range: ['625Mhz', '930Mhz']
   Power Profile Mode: 5-COMPUTE
   Power DPM Force Performance Level: manual
Ricks-Lab commented 4 years ago

Looks good! Should probably also make sure gpu-plot and gpu-mon --gui still works.

@KeithMyers Can you try the latest on master with your setup?

csecht commented 4 years ago

Yes, both are working. Nice! In monitor and plot, for the nvidia fields that can't be reported (VddGFX, P-states), do you want zero or something like, 'n/a', 'n/r', or 'unreadable' ? It's just that, at first glance, '0' can be misleading.

Ricks-Lab commented 4 years ago

The SCLK p-states should be valid, but NV's are opposite of AMD, where p-state 0 is highest performance. It is a little complicated to make VddGFX something other than numeric, since gpu-plot is a subprocess of `gpu-mon and communication is through a pipe. I will add to my list of things to do, but I will need to rethink some of the foundation for the plot utility.

KeithMyers commented 4 years ago

Rick, I just ran through the latest Beta-Master #4 from the most recent merge and all the apps ran fine on my Turing cards.

The only thing I could think of doing is to just ignore VddGFX somehow. But again, simple enough to just ignore it in the tables or toggle it off in plot too.

Ricks-Lab commented 4 years ago

I have attempted to address the vddgfx issue by using a numpy nan as the value. Can you give it a try?

KeithMyers commented 4 years ago

Sorry, that did not work. vddgfx is still present in the tables as it was before with a 0 value.

Also plot still has it present. The vddgfx button is still present and toggles the plot line off as before.

Ricks-Lab commented 4 years ago

Did you use the latest from master? If so, can you generate a debug log file? I don't see in the updated code where it is being set to zero.

KeithMyers commented 4 years ago

Yes, I used the latest from master. Only two hours since last commit. This commit https://github.com/Ricks-Lab/gpu-utils/commit/a89c0f586b73156820006e105a2d3abd64d5be64

DEBUG:gpu-utils:env.set_args:Command line arguments:
  Namespace(about=False, clinfo=False, debug=True, no_fan=False, ppm=False, pstates=False, short=False, table=False)
DEBUG:gpu-utils:env.set_args:Local TZ: PDT
DEBUG:gpu-utils:env.set_args:pciid path set to: /usr/share/misc/pci.ids
DEBUG:gpu-utils:env.set_args:Icon path set to: /home/keith/.local/share/rickslab-gpu-utils/icons
DEBUG:gpu-utils:gpu-ls.main:########## gpu-ls 3.4.11
DEBUG:gpu-utils:env.check_env:Using python: 3.8.2
DEBUG:gpu-utils:env.check_env:Using Linux Kernel: 5.4.0-37-generic
DEBUG:gpu-utils:env.check_env:Using Linux Distro: Ubuntu
DEBUG:gpu-utils:env.check_env:Linux Distro Description: Ubuntu 20.04 LTS
DEBUG:gpu-utils:env.check_env:Ubuntu package query tool: /usr/bin/dpkg
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [08:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2092515328]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:cl_index: {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [0a:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2091696128]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:cl_index: {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2091696128', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [0b:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2092515328]
DEBUG:gpu-utils:GPUmodule.set_gpu_list:OpenCL map: {'08:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}, '0a:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2091696128', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}, '0b:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}}
DEBUG:gpu-utils:env.read_amdfeaturemask:Could not read AMD Featuremask [[Errno 2] No such file or directory: '/sys/module/amdgpu/parameters/ppfeaturemask']
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 08:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 0a:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Found 3 GPUs
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item 5fa902384e50446aac226e81da3d62e7 to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 08:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['08:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card0/device] contents:
['uevent', 'resource3_wc', 'resource5', 'resource3', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'i2c-17', 'iommu_group', 'local_cpus', 'firmware_node', 'power', 'class', 'reset', 'i2c-15', 'numa_node', 'resource', 'rescan', 'max_link_width', 'msi_bus', 'device', 'i2c-13', 'boot_vga', 'aer_dev_nonfatal', 'current_link_width', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'i2c-18', 'resource1_wc', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'i2c-16', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'iommu', 'aer_dev_fatal', 'enable', 'link', 'i2c-14', 'modalias', 'i2c-12', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '08:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'model_short': 'UNKNOWN', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card0/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card0/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7982, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 0, GPU-089608fe-cba5-4711-bf68-085fd0711d8c, 100.83, 44, N/A, 2010, 2010, 7199, 1860, 2160, 2160, 7000, 100, 4, 1142, 100, [N/A], 4, 2, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '0', 'gpu_uuid': 'GPU-089608fe-cba5-4711-bf68-085fd0711d8c', 'power.draw': '100.83', 'temperature.gpu': '44', 'temperature.memory': 'N/A', 'clocks.gr': '2010', 'clocks.sm': '2010', 'clocks.mem': '7199', 'clocks.video': '1860', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '100', 'utilization.memory': '4', 'memory.used': '1142', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '4', 'pcie.link.gen.current': '2', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item 076a672142d94878abf73b72d12a75f0 to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 0a:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['0a:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card1/device] contents:
['uevent', 'resource3_wc', 'resource5', 'i2c-10', 'resource3', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'iommu_group', 'local_cpus', 'firmware_node', 'i2c-8', 'power', 'class', 'reset', 'numa_node', 'resource', 'rescan', 'i2c-6', 'max_link_width', 'msi_bus', 'device', 'boot_vga', 'aer_dev_nonfatal', 'current_link_width', 'i2c-11', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'resource1_wc', 'i2c-9', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'i2c-7', 'iommu', 'aer_dev_fatal', 'enable', 'link', 'i2c-5', 'modalias', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '0a:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'model_short': 'UNKNOWN', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card1/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card1/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7979, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 1, GPU-22b2c6ac-2d49-4863-197c-9c469071178a, 182.93, 44, N/A, 1980, 1980, 7199, 1830, 2160, 2160, 7000, 89, 57, 3765, 100, [N/A], 8, 3, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7979', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '1', 'gpu_uuid': 'GPU-22b2c6ac-2d49-4863-197c-9c469071178a', 'power.draw': '182.93', 'temperature.gpu': '44', 'temperature.memory': 'N/A', 'clocks.gr': '1980', 'clocks.sm': '1980', 'clocks.mem': '7199', 'clocks.video': '1830', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '89', 'utilization.memory': '57', 'memory.used': '3765', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item 77da83d87a2a4660ba76663e8171bf02 to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 0b:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card2/device] contents:
['uevent', 'resource3_wc', 'resource5', 'i2c-20', 'resource3', 'i2c-19', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'iommu_group', 'local_cpus', 'firmware_node', 'power', 'i2c-25', 'class', 'reset', 'numa_node', 'resource', 'rescan', 'max_link_width', 'msi_bus', 'i2c-23', 'device', 'boot_vga', 'aer_dev_nonfatal', 'i2c-21', 'current_link_width', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'resource1_wc', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'iommu', 'aer_dev_fatal', 'i2c-24', 'enable', 'link', 'i2c-22', 'modalias', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '0b:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'model_short': 'UNKNOWN', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card2/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card2/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7982, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 2, GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d, 202.45, 50, N/A, 1935, 1935, 7199, 1785, 2160, 2160, 7000, 97, 24, 305, 100, [N/A], 8, 3, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '2', 'gpu_uuid': 'GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d', 'power.draw': '202.45', 'temperature.gpu': '50', 'temperature.memory': 'N/A', 'clocks.gr': '1935', 'clocks.sm': '1935', 'clocks.mem': '7199', 'clocks.video': '1785', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '97', 'utilization.memory': '24', 'memory.used': '305', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7982, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 0, GPU-089608fe-cba5-4711-bf68-085fd0711d8c, 101.24, 44, N/A, 2010, 2010, 7199, 1860, 2160, 2160, 7000, 100, 4, 1142, 100, [N/A], 4, 2, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '0', 'gpu_uuid': 'GPU-089608fe-cba5-4711-bf68-085fd0711d8c', 'power.draw': '101.24', 'temperature.gpu': '44', 'temperature.memory': 'N/A', 'clocks.gr': '2010', 'clocks.sm': '2010', 'clocks.mem': '7199', 'clocks.video': '1860', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '100', 'utilization.memory': '4', 'memory.used': '1142', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '4', 'pcie.link.gen.current': '2', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7979, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 1, GPU-22b2c6ac-2d49-4863-197c-9c469071178a, 187.93, 44, N/A, 1980, 1980, 7199, 1830, 2160, 2160, 7000, 89, 57, 3765, 100, [N/A], 8, 3, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7979', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '1', 'gpu_uuid': 'GPU-22b2c6ac-2d49-4863-197c-9c469071178a', 'power.draw': '187.93', 'temperature.gpu': '44', 'temperature.memory': 'N/A', 'clocks.gr': '1980', 'clocks.sm': '1980', 'clocks.mem': '7199', 'clocks.video': '1830', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '89', 'utilization.memory': '57', 'memory.used': '3765', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.limit,power.min_limit,power.max_limit,memory.total,vbios_version,driver_version,compute_mode,name,serial,index,gpu_uuid,power.draw,temperature.gpu,temperature.memory,clocks.gr,clocks.sm,clocks.mem,clocks.video,clocks.max.gr,clocks.max.sm,clocks.max.mem,utilization.gpu,utilization.memory,memory.used,fan.speed,gom.current,pcie.link.width.current,pcie.link.gen.current,pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: [['200.00, 105.00, 292.00, 7982, 90.04.23.00.5F, 440.100, Default, GeForce RTX 2080, [N/A], 2, GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d, 202.45, 50, N/A, 1935, 1935, 7199, 1785, 2160, 2160, 7000, 97, 24, 305, 100, [N/A], 8, 3, P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '2', 'gpu_uuid': 'GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d', 'power.draw': '202.45', 'temperature.gpu': '50', 'temperature.memory': 'N/A', 'clocks.gr': '1935', 'clocks.sm': '1935', 'clocks.mem': '7199', 'clocks.video': '1785', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '97', 'utilization.memory': '24', 'memory.used': '305', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
KeithMyers commented 4 years ago 
┌─────────────┬────────────────┬────────────────┬────────────────┐
│Card #       │card0           │card1           │card2           │
├─────────────┼────────────────┼────────────────┼────────────────┤
│Model        │GeForce RTX 2080│GeForce RTX 2080│GeForce RTX 2080│
│GPU Load %   │100             │86              │96              │
│Mem Load %   │4               │68              │24              │
│VRAM Usage % │10.06           │47.299          │3.821           │
│GTT Usage %  │None            │None            │None            │
│Power (W)    │100.97          │190.97          │197.33          │
│Power Cap (W)│200.00          │200.00          │200.00          │
│Energy (kWh) │0.0             │0.0             │0.0             │
│T (C)        │44.0            │44.0            │50.0            │
│VddGFX (mV)  │0               │0               │0               │
│Fan Spd (%)  │100.0           │100.0           │100.0           │
│Sclk (MHz)   │2010            │1980            │1920            │
│Sclk Pstate  │2               │2               │2               │
│Mclk (MHz)   │7199            │7199            │7199            │
│Mclk Pstate  │2               │2               │2               │
│Perf Mode    │[N/A]           │[N/A]           │[N/A]           │
└─────────────┴────────────────┴────────────────┴────────────────┘
Ricks-Lab commented 4 years ago

Perhaps it is running the tool from your path instead of the one your local copy of the repository:

DEBUG:gpu-utils:gpu-ls.main:########## gpu-ls 3.4.11

Your debug log is showing version 3.4.11, but master is 3.4.12. Here is the first few lines of my debug log:

DEBUG:gpu-utils:env.set_args:Command line arguments:
  Namespace(about=False, clinfo=False, debug=True, no_fan=False, ppm=False, pstates=False, short=False, table=False)
DEBUG:gpu-utils:env.set_args:Local TZ: CST
DEBUG:gpu-utils:env.set_args:pciid path set to: /usr/share/misc/pci.ids
DEBUG:gpu-utils:env.set_args:Icon path set to: /home/rick/pydev/gpu-utils/GPUmodules/../icons
DEBUG:gpu-utils:gpu-ls.main:########## gpu-ls 3.4.12

I suggest to git clone from master with:

git clone https://github.com/Ricks-Lab/gpu-utils.git

To verify the version you are running execute gpu-ls --about

KeithMyers commented 4 years ago

OK, used git.

Author: RueiKe Copyright: Copyright (C) 2019 RicksLab Credits:
Craig Echt - Testing, Debug, Verification, and Documentation Keith Myers - Testing, Debug, Verification of NV Capability License: GNU General Public License Version: 3.4.12 Maintainer: RueiKe Status: Development Status :: 4 - Beta

Now I have nan for vddgfx in the table. And the vddgfx does not show up in plot now.

DEBUG:gpu-utils:env.set_args:Command line arguments:
  Namespace(about=False, clinfo=False, debug=True, no_fan=False, ppm=False, pstates=False, short=False, table=False)
DEBUG:gpu-utils:env.set_args:Local TZ: PDT
DEBUG:gpu-utils:env.set_args:pciid path set to: /usr/share/misc/pci.ids
DEBUG:gpu-utils:env.set_args:Icon path set to: /home/keith/Downloads/gpu-utils/GPUmodules/../icons
DEBUG:gpu-utils:gpu-ls.main:########## gpu-ls 3.4.12
DEBUG:gpu-utils:env.check_env:Using python: 3.8.2
DEBUG:gpu-utils:env.check_env:Using Linux Kernel: 5.4.0-37-generic
DEBUG:gpu-utils:env.check_env:Using Linux Distro: Ubuntu
DEBUG:gpu-utils:env.check_env:Linux Distro Description: Ubuntu 20.04 LTS
DEBUG:gpu-utils:env.check_env:Ubuntu package query tool: /usr/bin/dpkg
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [08:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2092515328]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:cl_index: {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [0a:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2091696128]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:cl_index: {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2091696128', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_NAME: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_VERSION: [OpenCL 1.2 CUDA]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DRIVER_VERSION: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_OPENCL_C_VERSION: [OpenCL C 1.2]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:NV ocl_pcie_id [0b:00.0]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_COMPUTE_UNITS: [46]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_ITEM_SIZES: [1024 1024 64]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_WORK_GROUP_SIZE: [1024]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE: [32]
DEBUG:gpu-utils:GPUmodule.read_gpu_opencl_data:openCL map CL_DEVICE_MAX_MEM_ALLOC_SIZE: [2092515328]
DEBUG:gpu-utils:GPUmodule.set_gpu_list:OpenCL map: {'08:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}, '0a:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2091696128', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}, '0b:00.0': {'prf_wg_multiple': '32', 'max_wg_size': '1024', 'prf_wg_size': None, 'max_wi_sizes': '1024 1024 64', 'max_wi_dim': '3', 'max_mem_allocation': '2092515328', 'simd_ins_width': None, 'simd_width': None, 'simd_per_cu': None, 'max_cu': '46', 'device_name': 'GeForce RTX 2080', 'opencl_version': 'OpenCL C 1.2', 'driver_version': '440.100', 'device_version': 'OpenCL 1.2 CUDA'}}
DEBUG:gpu-utils:env.read_amdfeaturemask:Could not read AMD Featuremask [[Errno 2] No such file or directory: '/sys/module/amdgpu/parameters/ppfeaturemask']
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 08:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 0a:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.get_gpu_pci_list:Found GPU pci: 0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Found 3 GPUs
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item 3437d7451f0e4b5c8d11b371c7b4ca72 to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 08:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['08:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card0/device] contents:
['uevent', 'resource3_wc', 'resource5', 'resource3', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'i2c-17', 'iommu_group', 'local_cpus', 'firmware_node', 'power', 'class', 'reset', 'i2c-15', 'numa_node', 'resource', 'rescan', 'max_link_width', 'msi_bus', 'device', 'i2c-13', 'boot_vga', 'aer_dev_nonfatal', 'current_link_width', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'i2c-18', 'resource1_wc', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'i2c-16', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'iommu', 'aer_dev_fatal', 'enable', 'link', 'i2c-14', 'modalias', 'i2c-12', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '08:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card0/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card0/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item df3c1c89015f4441b2c8c9551f69795e to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 0a:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['0a:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card1/device] contents:
['uevent', 'resource3_wc', 'resource5', 'i2c-10', 'resource3', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'iommu_group', 'local_cpus', 'firmware_node', 'i2c-8', 'power', 'class', 'reset', 'numa_node', 'resource', 'rescan', 'i2c-6', 'max_link_width', 'msi_bus', 'device', 'boot_vga', 'aer_dev_nonfatal', 'current_link_width', 'i2c-11', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'resource1_wc', 'i2c-9', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'i2c-7', 'iommu', 'aer_dev_fatal', 'enable', 'link', 'i2c-5', 'modalias', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '0a:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card1/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card1/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.add:Added GPU Item b1fd1aecf1cf47e29e345cbedebc95de to GPU List
DEBUG:gpu-utils:GPUmodule.set_gpu_list:GPU: 0b:00.0
DEBUG:gpu-utils:GPUmodule.set_gpu_list:lspci output items:
 ['0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', '\tSubsystem: eVga.com. Corp. TU104 [GeForce RTX 2080 Rev. A]', '\tKernel driver in use: nvidia', '\tKernel modules: nvidiafb, nouveau, nvidia_drm, nvidia', '']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.1/0000:0a:00.0
device_dir: /sys/class/drm/card1/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0
device_dir: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:card_path set to: /sys/class/drm/card2/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:sysfpath: /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.2/0000:03:04.0/0000:08:00.0
device_dir: /sys/class/drm/card0/device
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card dir [/sys/class/drm/card2/device] contents:
['uevent', 'resource3_wc', 'resource5', 'i2c-20', 'resource3', 'i2c-19', 'broken_parity_status', 'subsystem_device', 'rom', 'dma_mask_bits', 'vendor', 'resource1', 'iommu_group', 'local_cpus', 'firmware_node', 'power', 'i2c-25', 'class', 'reset', 'numa_node', 'resource', 'rescan', 'max_link_width', 'msi_bus', 'i2c-23', 'device', 'boot_vga', 'aer_dev_nonfatal', 'i2c-21', 'current_link_width', 'driver', 'max_link_speed', 'local_cpulist', 'driver_override', 'subsystem', 'd3cold_allowed', 'irq', 'revision', 'current_link_speed', 'resource1_wc', 'aer_dev_correctable', 'consistent_dma_mask_bits', 'resource0', 'config', 'ari_enabled', 'msi_irqs', 'remove', 'iommu', 'aer_dev_fatal', 'i2c-24', 'enable', 'link', 'i2c-22', 'modalias', 'subsystem_vendor', 'drm']
DEBUG:gpu-utils:GPUmodule.set_gpu_list:HW file search: []
DEBUG:gpu-utils:GPUmodule.populate_prm_from_dict:prm dict:
{'pcie_id': '0b:00.0', 'model': 'NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1)', 'vendor': <vendor.NVIDIA: 4>, 'driver': 'nvidiafb, nouveau, nvidia_drm, nvidia', 'card_path': '/sys/class/drm/card2/device', 'sys_card_path': '/sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0', 'gpu_type': <type.Supported: 3>, 'hwmon_path': '', 'readable': True, 'writable': False, 'compute': True, 'compute_platform': 'OpenCL 1.2 CUDA'}
DEBUG:gpu-utils:GPUmodule.set_gpu_list:Card flags: readable: True, writable: False, type: Supported
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_generic:sensor path set to [/sys/class/drm/card2/device]
DEBUG:gpu-utils:GPUmodule.read_pciid_model:Logger active in module
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['200.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.limit], result: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.min_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['105.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.min_limit], result: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.max_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['292.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.max_limit], result: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=memory.total --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7982', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7982]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.total], result: [7982]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=vbios_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['90.04.23.00.5F', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [vbios_version], result: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=driver_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['440.100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [driver_version], result: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=compute_mode --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['Default', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [compute_mode], result: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=name --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GeForce RTX 2080', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [name], result: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=serial --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [serial], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=index --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['0', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [0]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [index], result: [0]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=gpu_uuid --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GPU-089608fe-cba5-4711-bf68-085fd0711d8c', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GPU-089608fe-cba5-4711-bf68-085fd0711d8c]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gpu_uuid], result: [GPU-089608fe-cba5-4711-bf68-085fd0711d8c]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=power.draw --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['100.45', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [100.45]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.draw], result: [100.45]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=temperature.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['42', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [42]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.gpu], result: [42]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=temperature.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['N/A', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.memory], result: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2010', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2010]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.gr], result: [2010]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2010', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2010]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.sm], result: [2010]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7199', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.mem], result: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.video --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1860', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1860]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.video], result: [1860]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.max.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.gr], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.max.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.sm], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=clocks.max.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7000', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.mem], result: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=utilization.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.gpu], result: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=utilization.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['4', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [4]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.memory], result: [4]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=memory.used --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1029', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1029]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.used], result: [1029]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=fan.speed --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [fan.speed], result: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=gom.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gom.current], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=pcie.link.width.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['4', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [4]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.width.current], result: [4]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=pcie.link.gen.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.gen.current], result: [2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 08:00.0 --query-gpu=pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pstate], result: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '0', 'gpu_uuid': 'GPU-089608fe-cba5-4711-bf68-085fd0711d8c', 'power.draw': '100.45', 'temperature.gpu': '42', 'temperature.memory': 'N/A', 'clocks.gr': '2010', 'clocks.sm': '2010', 'clocks.mem': '7199', 'clocks.video': '1860', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '100', 'utilization.memory': '4', 'memory.used': '1029', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '4', 'pcie.link.gen.current': '2', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['200.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.limit], result: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.min_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['105.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.min_limit], result: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.max_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['292.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.max_limit], result: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=memory.total --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7979', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7979]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.total], result: [7979]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=vbios_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['90.04.23.00.5F', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [vbios_version], result: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=driver_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['440.100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [driver_version], result: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=compute_mode --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['Default', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [compute_mode], result: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=name --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GeForce RTX 2080', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [name], result: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=serial --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [serial], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=index --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [index], result: [1]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=gpu_uuid --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GPU-22b2c6ac-2d49-4863-197c-9c469071178a', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GPU-22b2c6ac-2d49-4863-197c-9c469071178a]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gpu_uuid], result: [GPU-22b2c6ac-2d49-4863-197c-9c469071178a]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=power.draw --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['163.50', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [163.50]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.draw], result: [163.50]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=temperature.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['40', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [40]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.gpu], result: [40]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=temperature.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['N/A', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.memory], result: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1980', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1980]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.gr], result: [1980]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1980', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1980]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.sm], result: [1980]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7199', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.mem], result: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.video --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1830', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1830]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.video], result: [1830]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.max.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.gr], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.max.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.sm], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=clocks.max.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7000', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.mem], result: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=utilization.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['98', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [98]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.gpu], result: [98]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=utilization.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['43', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [43]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.memory], result: [43]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=memory.used --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1242', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1242]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.used], result: [1242]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=fan.speed --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [fan.speed], result: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=gom.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gom.current], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=pcie.link.width.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['8', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [8]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.width.current], result: [8]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=pcie.link.gen.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['3', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.gen.current], result: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0a:00.0 --query-gpu=pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pstate], result: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7979', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '1', 'gpu_uuid': 'GPU-22b2c6ac-2d49-4863-197c-9c469071178a', 'power.draw': '163.50', 'temperature.gpu': '40', 'temperature.memory': 'N/A', 'clocks.gr': '1980', 'clocks.sm': '1980', 'clocks.mem': '7199', 'clocks.video': '1830', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '98', 'utilization.memory': '43', 'memory.used': '1242', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['200.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.limit], result: [200.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.min_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['105.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.min_limit], result: [105.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.max_limit --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['292.00', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.max_limit], result: [292.00]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=memory.total --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7982', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7982]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.total], result: [7982]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=vbios_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['90.04.23.00.5F', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [vbios_version], result: [90.04.23.00.5F]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=driver_version --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['440.100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [driver_version], result: [440.100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=compute_mode --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['Default', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [compute_mode], result: [Default]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=name --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GeForce RTX 2080', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [name], result: [GeForce RTX 2080]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=serial --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [serial], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=index --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [index], result: [2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=gpu_uuid --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gpu_uuid], result: [GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=power.draw --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['194.95', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [194.95]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [power.draw], result: [194.95]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=temperature.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['48', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [48]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.gpu], result: [48]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=temperature.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['N/A', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [temperature.memory], result: [N/A]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1920', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1920]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.gr], result: [1920]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1935', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1935]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.sm], result: [1935]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7199', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.mem], result: [7199]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.video --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['1785', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [1785]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.video], result: [1785]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.max.gr --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.gr], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.max.sm --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['2160', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.sm], result: [2160]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=clocks.max.mem --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['7000', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [clocks.max.mem], result: [7000]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=utilization.gpu --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['97', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [97]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.gpu], result: [97]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=utilization.memory --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['24', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [24]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [utilization.memory], result: [24]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=memory.used --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['307', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [307]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [memory.used], result: [307]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=fan.speed --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['100', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [fan.speed], result: [100]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=gom.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['[N/A]', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [gom.current], result: [[N/A]]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=pcie.link.width.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['8', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [8]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.width.current], result: [8]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=pcie.link.gen.current --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['3', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pcie.link.gen.current], result: [3]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV command:
/usr/bin/nvidia-smi -i 0b:00.0 --query-gpu=pstate --format=csv,noheader,nounits
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query result: [['P2', '']]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_nv:NV query response: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query (each-call) query item [pstate], result: [P2]
DEBUG:gpu-utils:GPUmodule.read_gpu_sensor_set_nv:NV query result: {'power.limit': '200.00', 'power.min_limit': '105.00', 'power.max_limit': '292.00', 'memory.total': '7982', 'vbios_version': '90.04.23.00.5F', 'driver_version': '440.100', 'compute_mode': 'Default', 'name': 'GeForce RTX 2080', 'serial': '[N/A]', 'index': '2', 'gpu_uuid': 'GPU-6933bcb7-7072-2181-a92f-0e5c42c3fa0d', 'power.draw': '194.95', 'temperature.gpu': '48', 'temperature.memory': 'N/A', 'clocks.gr': '1920', 'clocks.sm': '1935', 'clocks.mem': '7199', 'clocks.video': '1785', 'clocks.max.gr': '2160', 'clocks.max.sm': '2160', 'clocks.max.mem': '7000', 'utilization.gpu': '97', 'utilization.memory': '24', 'memory.used': '307', 'fan.speed': '100', 'gom.current': '[N/A]', 'pcie.link.width.current': '8', 'pcie.link.gen.current': '3', 'pstate': 'P2'}
Ricks-Lab commented 4 years ago

Is gpu-ls noticeably slower now that it reads one sensor at a time for the first read?

@csecht I will release it after you have a chance to check it out on your system.

KeithMyers commented 4 years ago

Yes, I would guess maybe 2X as long to read the cards on initial running compared to the previous version.

But on my system that only amounts to about another second. Not worried about it.

csecht commented 4 years ago

On my system it is only a slight stutter, which I hadn't really noticed before. For some reason, when running the old NVIDIA card on E@H pulsar #1 tasks, my Firefox browser really slows down. To browse sites, I need to suspend tasks. gpu-utils runs well, though not as zippy as previously.

Ricks-Lab commented 4 years ago

The extra one time reads are only for NV as the mechanism for AMD always required this. The first read will disable unreadable sensors, so the additional reads by plot and mon will just read valid sensors.

Have you checked out if the vddgfx fixes look good on your system?

csecht commented 4 years ago

Yes, that works well and looks good. Good to know about NVIDIA p-states having 0 as highest. The text for vddgfx helps, but, as an aside, what does 'nan' stand for?

Ricks-Lab commented 4 years ago

It stands for not a number. It is a datatype used in numpy to indicate a float value that can not be described with a numeric value, like infinite.

Ricks-Lab commented 4 years ago

I have one more change before I call this release complete. For the first sensor read, I will classify the card as unreadable if no sensors can be read. I will request verification here when it is complete.

Ricks-Lab commented 4 years ago

I just pushed 3.4.13 which will set NV card readability to False if no parameters can be read. Let me know if you see any issues with the latest on master.

csecht commented 4 years ago

Everythings works, except what is noted below.

Found a possible bug, something I thought was working a while back, but am not sure. Now, however, clinfo data does not show for the RX 5600 XT card, but does for NVIDIA on the same host (debug file attached). clinfo does show for both two RX 570 cards on my other host. 

$ ./gpu-ls --clinfo
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: INTEL: 1, NVIDIA: 1, AMD: 1
AMD: amdgpu version: 20.10-1048554
AMD: Wattman features enabled: 0xfffd7fff
3 total GPUs, 1 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor: INTEL
   Readable: False
   Writable: False
   Compute: False
   Device ID: {'device': '0x3e91', 'subsystem_device': '0x8694', 'subsystem_vendor': '0x1043', 'vendor': '0x8086'}
   Decoded Device ID: 8th Gen Core Processor Gaussian Mixture Model
   Card Model: Intel Corporation 8th Gen Core Processor Gaussian Mixture Model
   PCIe ID: 00:02.0
   Driver: i915
   GPU Type: Unsupported
   HWmon: None
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:02.0

Card Number: 1
   Vendor: NVIDIA
   Readable: True
   Writable: False
   Compute: True
   GPU UID: GPU-fcbaadc4-4040-c2e5-d5b6-52d1547bcc64
   GPU S/N: [Not Supported]
   Device ID: {'device': '0x1381', 'subsystem_device': '0x1073', 'subsystem_vendor': '0x10de', 'vendor': '0x10de'}
   Decoded Device ID: GM107 [GeForce GTX 750]
   Card Model: GeForce GTX 750
   Display Card Model: GeForce GTX 750
   Card Index: 0
   PCIe ID: 01:00.0
      Link Speed: GEN3
      Link Width: 8
   ##################################################
   Driver: 390.138
   vBIOS Version: 82.07.32.00.32
   Compute Platform: OpenCL 1.2 CUDA
   Compute Mode: Default
   GPU Type: Supported
   HWmon: None
   Card Path: /sys/class/drm/card1/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0
   ##################################################
   Current Power (W): 17.180
   Power Cap (W): 38.50
      Power Cap Range (W): [30.0, 38.5]
   Fan Target Speed (rpm): None
   Current Fan PWM (%): 40.000
   ##################################################
   Current GPU Loading (%): 100
   Current Memory Loading (%): 34
   Current VRAM Usage (%): 90.214
      Current VRAM Used (GB): 0.864
      Total VRAM (GB): 0.958
   Current  Temps (C): {'temperature.gpu': 47.0, 'temperature.memory': None}
   Current Clk Frequencies (MHz): {'clocks.gr': 1163.0, 'clocks.mem': 2505.0, 'clocks.sm': 1163.0, 'clocks.video': 1046.0}
   Maximum Clk Frequencies (MHz): {'clocks.max.gr': 1293.0, 'clocks.max.mem': 2505.0, 'clocks.max.sm': 1293.0}
   Current SCLK P-State: [0, '']
   Power Profile Mode: [Not Supported]
   ##################################################
   Device OpenCL C Version: OpenCL C 1.2
   Device Name: GeForce GTX 750
   Device Version: OpenCL 1.2 CUDA
   Driver Version: 390.138
   Max Compute Units: 4
   SIMD per CU: 2
   SIMD Width: 32
   SIMD Instruction Width: 1
   CL Max Memory Allocation: 257261568
   Max Work Item Dimensions: 3
   Max Work Item Sizes: 1024 1024 64
   Max Work Group Size: 1024
   Preferred Work Group Size: 256
   Preferred Work Group Multiple: 32

Card Number: 2
   Vendor: AMD
   Readable: True
   Writable: True
   Compute: False
   GPU UID: None
   Device ID: {'device': '0x731f', 'subsystem_device': '0xe411', 'subsystem_vendor': '0x1da2', 'vendor': '0x1002'}
   Decoded Device ID: Radeon RX 5600 XT
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev ca)
   Display Card Model: Radeon RX 5600 XT
   PCIe ID: 04:00.0
      Link Speed: 16 GT/s
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-5E4111U-X4G
   Compute Platform: None
   GPU Type: CurvePts
   HWmon: /sys/class/drm/card2/device/hwmon/hwmon3
   Card Path: /sys/class/drm/card2/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/0000:03:00.0/0000:04:00.0
   ##################################################
   Current Power (W): 117.000
   Power Cap (W): 160.000
      Power Cap Range (W): [0, 192]
   Fan Enable: 0
   Fan PWM Mode: [2, 'Dynamic']
   Fan Target Speed (rpm): 1026
   Current Fan Speed (rpm): 1026
   Current Fan PWM (%): 31
      Fan Speed Range (rpm): [0, 3200]
      Fan PWM Range (%): [0, 100]
   ##################################################
   Current GPU Loading (%): 92
   Current Memory Loading (%): 37
   Current GTT Memory Usage (%): 0.432
      Current GTT Memory Used (GB): 0.026
      Total GTT Memory (GB): 5.984
   Current VRAM Usage (%): 11.969
      Current VRAM Used (GB): 0.716
      Total VRAM (GB): 5.984
   Current  Temps (C): {'edge': 59.0, 'junction': 67.0, 'mem': 82.0}
   Critical Temps (C): {'edge': 118.0, 'junction': 99.0, 'mem': 99.0}
   Current Voltages (V): {'vddgfx': 887}
   Current Clk Frequencies (MHz): {'mclk': 875.0, 'sclk': 1700.0}
   Current SCLK P-State: [2, '1700Mhz']
      SCLK Range: ['800Mhz', '1820Mhz']
   Current MCLK P-State: [3, '875Mhz']
      MCLK Range: ['625Mhz', '930Mhz']
   Power Profile Mode: 0-BOOTUP_DEFAULT
   Power DPM Force Performance Level: manual

This may be related to the clinfo issue; Why does the Navi10 card show Compute: False, when it does run the COMPUTE PPM? The NVIDIA and the RX570 cards all show Compute: True.

Card Number: 2
   Vendor: AMD
   Readable: True
   Writable: True
   Compute: False
   Device ID: {'device': '0x731f', 'subsystem_device': '0xe411', 'subsystem_vendor': '0x1da2', 'vendor': '0x1002'}
   Decoded Device ID: Radeon RX 5600 XT
   PCIe ID: 04:00.0
   GPU Type: CurvePts
   HWmon: /sys/class/drm/card2/device/hwmon/hwmon3
   Card Path: /sys/class/drm/card2/device
   System Card Path: /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/0000:03:00.0/0000:04:00.0

debug_gpu-ls_clinfo.txt

csecht commented 4 years ago

I can work on updating the User Guide tomorrow. Which examples and screenshots do you want to show the hybrid AMD/NV system?

Ricks-Lab commented 4 years ago

Can you verify if it is possibly using an nvidia specific clinfo? which clinfo

And use the clinfo command line to see if the AMD card shows up?

Ricks-Lab commented 4 years ago

Maybe there is something that I did not anticipate in an AMD/NV system. Can you provide the output of clinfo --raw

Ricks-Lab commented 4 years ago

I can work on updating the User Guide tomorrow. Which examples and screenshots do you want to show the hybrid AMD/NV system?

I think it would be good to demonstrate the increased compatibility where ever possible. If there is some point that needs to made with another example that is only AMD, I think that should be prioritized over showing the hybrid setup. I think at least the plot image is necessary to make the point.

Ricks-Lab commented 4 years ago

This may be related to the clinfo issue; Why does the Navi10 card show Compute: False, when it does run the COMPUTE PPM? The NVIDIA and the RX570 cards all show Compute: True.

Compute capability is determined from the output of clinfo --raw. The ppm compute mode is only a performance mode that impacts how the card changes performance states and does not influence compute mode being available or not. Getting the clinfo --raw output will be critical in understanding the issue.

Ricks-Lab commented 4 years ago

I think I have found the issue and pushed a fix. I found that the utility was only looking at the clinfo index, but the index starts from zero for each vendor. Now it looks at both vendor and index.

csecht commented 4 years ago

Yes, good, that fixed the clinfo option for the AMD card and its compute true status. gpu-ls --clinfo now works for all cards in:

Version:  3.4.13
Maintainer:  RueiKe
Status:  Development Status :: 4 - Beta
Ricks-Lab commented 4 years ago

I have released the latest to PyPI!

Ricks-Lab commented 4 years ago

@csecht I have merged the pull request for the User Guide updates. My only concern is that it looks like long model names are not truncated in gpu-pac. I have included truncation so that the interface works with more cards. Let me know of any concerns. Also, it would be good to change the pac image since it still indicates the old amdgpu name.

csecht commented 4 years ago

That PAC screenshot I used was from before pci-id database updated (with the request I had made earlier for that Navi10 card). By the time the updated pci-id displayed the short model name (Radeon RX 5600 XT), I had taken the RX 570 card out of that machine and so lost the opportunity to show both cards types with their short model names in the one PAC window. So we can leave that slightly outdated PAC graphic with the long name as is or replace it with two separate PAC screenshots for each card type (with their short names). It’s good though to use truncation, because no doubt there are other cards without short model names in the database.

On Jul 5, 2020, at 7:57 PM, Rick notifications@github.com wrote:

@csecht https://github.com/csecht I have merged the pull request https://github.com/Ricks-Lab/gpu-utils/pull/93 for the User Guide updates. My only concern is that it looks like long model names are not truncated in gpu-pac. I have included truncation so that the interface works with more cards. Let me know of any concerns.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Ricks-Lab/gpu-utils/issues/88#issuecomment-653965039, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALMVCQULSKHL62VDQKTYGVDR2EOO7ANCNFSM4OGEFXNA.

Ricks-Lab commented 4 years ago

Thanks for the explanation. Maybe it is best to leave it as it is until either of us has a better setup that demonstrates gpu-util capability.

Ricks-Lab commented 4 years ago

Officially released v3.5.0!