search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.87k stars 5.6k forks source link

Can you collect GPU monitoring for each process consumed #5546

Closed taoxuFreedomUP closed 8 months ago

taoxuFreedomUP commented 5 years ago

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Current behavior:

Desired behavior:

Use case: [Why is this important (helps with prioritizing requests)]

| Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1110 C /usr/local/jdk/jdk1.8.0_162/bin/java 2727MiB | | 0 2211 C /usr/local/jdk/jdk1.8.0_162/bin/java 2727MiB | | 0 183754 C /usr/local/jdk/jdk1.8.0_162/bin/java 129MiB | +-----------------------------------------------------------------------------+

powersj commented 2 years ago

Hi,

Sorry no one has gotten back to you yet.

From the output it looks like you are using the nvidia-smi plugin. Based on the output of /usr/bin/nvidia-smi -q -x command that telegraf runs it does look like process info is collected and looks like this:

            <process_info>
                <gpu_instance_id>N/A</gpu_instance_id>
                <compute_instance_id>N/A</compute_instance_id>
                <pid>3085</pid>
                <type>G</type>
                <process_name>/usr/lib/firefox/firefox</process_name>
                <used_memory>261 MiB</used_memory>
            </process_info>
            <process_info>
                <gpu_instance_id>N/A</gpu_instance_id>
                <compute_instance_id>N/A</compute_instance_id>
                <pid>3371</pid>
                <type>G</type>
                <process_name>/usr/lib/firefox/firefox</process_name>
                <used_memory>3 MiB</used_memory>
            </process_info>
            <process_info>
                <gpu_instance_id>N/A</gpu_instance_id>
                <compute_instance_id>N/A</compute_instance_id>
                <pid>147337</pid>
                <type>G</type>
                <process_name>/usr/lib/firefox/firefox</process_name>
                <used_memory>3 MiB</used_memory>
            </process_info>

Based on that data, what are you interested in seeing get collected?

Thanks!

telegraf-tiger[bot] commented 2 years ago

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!

EuleMitKeule commented 8 months ago

Can we reopen this? It would be gread to get the list of processes that are using the GPU as a metric, including PID, name and memory.

powersj commented 8 months ago

@EuleMitKeule please test the artifacts in #15023 and let us know if it resolves the request.

EuleMitKeule commented 8 months ago

@powersj I haven't had time to try it out, but I'll report back when I did. Thank you very much for implementing this!

EuleMitKeule commented 8 months ago

@powersj It works great, thanks! However I would love for the process name also to be available, so the user knows what application uses the GPU without having to look up the PID. Would be awesome if this could also be included!

EuleMitKeule commented 8 months ago

Also maybe it would make more sense to have only a single metric nvidia_smi_process_memory that has a process_name and process_id label so everything can be easily displayed at once in a table or time series? I am using Telegraf with Prometheus and Grafana in case that isn't clear.

powersj commented 8 months ago

@EuleMitKeule,

The plugin collects the PID, name, memory, and type already. You'll want to convert the PID to a tag so it shows up as a label in Prometheus. That should get you what you are after. Let me know if that helps.

Something like this untested config:

[[processors.converter]]
  namepass = ["nvidia_smi_process"]
  [processors.converter.fields]
    tag = ["pid"]
EuleMitKeule commented 8 months ago

@powersj This does work as you explained:

[[processors.converter]]
  namepass = ["nvidia_smi_process"]
  [processors.converter.fields]
    tag = ["pid"]

This does not, there is no process_name label

[[processors.converter]]
  namepass = ["nvidia_smi_process"]
  [processors.converter.fields]
    tag = ["process_name"]

I only have the metrics nvidia_smi_process_pid and nvidia_smi_process_used_memory and they only have the type label and pid with the additional config above. I am not able to get the process_name. I also tried name, which also did not work.

powersj commented 8 months ago

You should not need to convert the name, it is already a tag and should already be a label. Please provide some example metrics that you are seeing.

On my system I get the following:

# HELP nvidia_smi_process_used_memory Telegraf collected metric
# TYPE nvidia_smi_process_used_memory untyped
nvidia_smi_process_used_memory{name="/usr/lib/Xorg",pid="819",type="G"} 412
# HELP nvidia_smi_process_used_memory Telegraf collected metric
# TYPE nvidia_smi_process_used_memory untyped
nvidia_smi_process_used_memory{name="/usr/bin/gnome-shell",pid="1483",type="G"} 27

with this config:

[[outputs.file]]
  data_format = "prometheus"

[[inputs.nvidia_smi]]

[[processors.converter]]
  [processors.converter.fields]
    tag = ["pid"]
EuleMitKeule commented 8 months ago 
# HELP nvidia_smi_process_pid Telegraf collected metric
# TYPE nvidia_smi_process_pid untyped
nvidia_smi_process_pid{host="docker",type="C"} 1.713717e+06
# HELP nvidia_smi_process_used_memory Telegraf collected metric
# TYPE nvidia_smi_process_used_memory untyped
nvidia_smi_process_used_memory{host="docker",type="C"} 203

Telegraf is running in a docker container. Host output for nvidia_smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro P2000                   Off |   00000000:00:10.0 Off |                  N/A |
| 58%   59C    P0             29W /   75W |     205MiB /   5120MiB |     11%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1713717      C   /usr/lib/jellyfin-ffmpeg/ffmpeg               203MiB |
+-----------------------------------------------------------------------------------------+

docker exec -it telegraf nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro P2000                   Off |   00000000:00:10.0 Off |                  N/A |
| 58%   59C    P0             32W /   75W |     205MiB /   5120MiB |     11%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

docker-compose.yaml

  telegraf:
    <<: *x-defaults
    image: quay.io/influxdb/telegraf-nightly:latest
    container_name: telegraf
    privileged: true
    network_mode: host
    environment:
      - HOST_PROC=/rootfs/proc
      - HOST_SYS=/rootfs/sys
      - HOST_ETC=/rootfs/etc
      - HOST_MOUNT_PREFIX=/rootfs
      - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so
    runtime: nvidia
    devices:
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia0:/dev/nvidia0
    volumes:
      - /var/run/utmp:/var/run/utmp:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/rootfs:ro
      - /sys:/rootfs/sys:ro
      - /etc:/rootfs/etc:ro
      - /proc:/rootfs/proc:ro
      - /run/udev:/run/udev:ro
      - /mnt/appdata/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro
      - /usr/lib/x86_64-linux-gnu/libnvidia-ml.so:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so:ro
powersj commented 8 months ago

Can you provide the raw output via:

nvidia-smi -x -q

as well as the output of the metrics in line protocol format so we can avoid any prometheus-ism by adding the following output:

[[outputs.file]]

That will print the line protocol metrics to stdout.

EuleMitKeule commented 8 months ago

docker exec -it telegraf nvidia-smi -x -q:

<?xml version="1.0" ?>
<!DOCTYPE nvidia_smi_log SYSTEM "nvsmi_device_v12.dtd">
<nvidia_smi_log>
        <timestamp>Mon Mar 25 14:41:34 2024</timestamp>
        <driver_version>550.54.14</driver_version>
        <cuda_version>12.4</cuda_version>
        <attached_gpus>1</attached_gpus>
        <gpu id="00000000:00:10.0">
                <product_name>Quadro P2000</product_name>
                <product_brand>Quadro</product_brand>
                <product_architecture>Pascal</product_architecture>
                <display_mode>Disabled</display_mode>
                <display_active>Disabled</display_active>
                <persistence_mode>Disabled</persistence_mode>
                <addressing_mode>N/A</addressing_mode>
                <mig_mode>
                        <current_mig>N/A</current_mig>
                        <pending_mig>N/A</pending_mig>
                </mig_mode>
                <mig_devices>
                        None
                </mig_devices>
                <accounting_mode>Disabled</accounting_mode>
                <accounting_mode_buffer_size>4000</accounting_mode_buffer_size>
                <driver_model>
                        <current_dm>N/A</current_dm>
                        <pending_dm>N/A</pending_dm>
                </driver_model>
                <serial>0320218076820</serial>
                <uuid>GPU-68be6701-ee4c-5659-9384-45319d269070</uuid>
                <minor_number>0</minor_number>
                <vbios_version>86.06.3F.00.30</vbios_version>
                <multigpu_board>No</multigpu_board>
                <board_id>0x10</board_id>
                <board_part_number>900-5G410-0100-000</board_part_number>
                <gpu_part_number>1C30-875-A1</gpu_part_number>
                <gpu_fru_part_number>N/A</gpu_fru_part_number>
                <gpu_module_id>1</gpu_module_id>
                <inforom_version>
                        <img_version>G410.0502.00.02</img_version>
                        <oem_object>1.1</oem_object>
                        <ecc_object>N/A</ecc_object>
                        <pwr_object>N/A</pwr_object>
                </inforom_version>
                <inforom_bbx_flush>
                        <latest_timestamp>N/A</latest_timestamp>
                        <latest_duration>N/A</latest_duration>
                </inforom_bbx_flush>
                <gpu_operation_mode>
                        <current_gom>N/A</current_gom>
                        <pending_gom>N/A</pending_gom>
                </gpu_operation_mode>
                <c2c_mode>N/A</c2c_mode>
                <gpu_virtualization_mode>
                        <virtualization_mode>Pass-Through</virtualization_mode>
                        <host_vgpu_mode>N/A</host_vgpu_mode>
                        <vgpu_heterogeneous_mode>N/A</vgpu_heterogeneous_mode>
                </gpu_virtualization_mode>
                <gpu_reset_status>
                        <reset_required>No</reset_required>
                        <drain_and_reset_recommended>N/A</drain_and_reset_recommended>
                </gpu_reset_status>
                <gsp_firmware_version>N/A</gsp_firmware_version>
                <ibmnpu>
                        <relaxed_ordering_mode>N/A</relaxed_ordering_mode>
                </ibmnpu>
                <pci>
                        <pci_bus>00</pci_bus>
                        <pci_device>10</pci_device>
                        <pci_domain>0000</pci_domain>
                        <pci_base_class>3</pci_base_class>
                        <pci_sub_class>0</pci_sub_class>
                        <pci_device_id>1C3010DE</pci_device_id>
                        <pci_bus_id>00000000:00:10.0</pci_bus_id>
                        <pci_sub_system_id>11B31028</pci_sub_system_id>
                        <pci_gpu_link_info>
                                <pcie_gen>
                                        <max_link_gen>3</max_link_gen>
                                        <current_link_gen>3</current_link_gen>
                                        <device_current_link_gen>3</device_current_link_gen>
                                        <max_device_link_gen>3</max_device_link_gen>
                                        <max_host_link_gen>N/A</max_host_link_gen>
                                </pcie_gen>
                                <link_widths>
                                        <max_link_width>16x</max_link_width>
                                        <current_link_width>8x</current_link_width>
                                </link_widths>
                        </pci_gpu_link_info>
                        <pci_bridge_chip>
                                <bridge_chip_type>N/A</bridge_chip_type>
                                <bridge_chip_fw>N/A</bridge_chip_fw>
                        </pci_bridge_chip>
                        <replay_counter>0</replay_counter>
                        <replay_rollover_counter>0</replay_rollover_counter>
                        <tx_util>17 KB/s</tx_util>
                        <rx_util>34 KB/s</rx_util>
                        <atomic_caps_inbound>N/A</atomic_caps_inbound>
                        <atomic_caps_outbound>N/A</atomic_caps_outbound>
                </pci>
                <fan_speed>58 %</fan_speed>
                <performance_state>P0</performance_state>
                <clocks_event_reasons>
                        <clocks_event_reason_gpu_idle>Not Active</clocks_event_reason_gpu_idle>
                        <clocks_event_reason_applications_clocks_setting>Not Active</clocks_event_reason_applications_clocks_setting>
                        <clocks_event_reason_sw_power_cap>Not Active</clocks_event_reason_sw_power_cap>
                        <clocks_event_reason_hw_slowdown>Not Active</clocks_event_reason_hw_slowdown>
                        <clocks_event_reason_hw_thermal_slowdown>Not Active</clocks_event_reason_hw_thermal_slowdown>
                        <clocks_event_reason_hw_power_brake_slowdown>Not Active</clocks_event_reason_hw_power_brake_slowdown>
                        <clocks_event_reason_sync_boost>Not Active</clocks_event_reason_sync_boost>
                        <clocks_event_reason_sw_thermal_slowdown>Not Active</clocks_event_reason_sw_thermal_slowdown>
                        <clocks_event_reason_display_clocks_setting>Not Active</clocks_event_reason_display_clocks_setting>
                </clocks_event_reasons>
                <sparse_operation_mode>N/A</sparse_operation_mode>
                <fb_memory_usage>
                        <total>5120 MiB</total>
                        <reserved>73 MiB</reserved>
                        <used>205 MiB</used>
                        <free>4840 MiB</free>
                </fb_memory_usage>
                <bar1_memory_usage>
                        <total>256 MiB</total>
                        <used>2 MiB</used>
                        <free>254 MiB</free>
                </bar1_memory_usage>
                <cc_protected_memory_usage>
                        <total>0 MiB</total>
                        <used>0 MiB</used>
                        <free>0 MiB</free>
                </cc_protected_memory_usage>
                <compute_mode>Default</compute_mode>
                <utilization>
                        <gpu_util>9 %</gpu_util>
                        <memory_util>12 %</memory_util>
                        <encoder_util>45 %</encoder_util>
                        <decoder_util>45 %</decoder_util>
                        <jpeg_util>N/A</jpeg_util>
                        <ofa_util>N/A</ofa_util>
                </utilization>
                <encoder_stats>
                        <session_count>1</session_count>
                        <average_fps>426</average_fps>
                        <average_latency>2246</average_latency>
                </encoder_stats>
                <fbc_stats>
                        <session_count>0</session_count>
                        <average_fps>0</average_fps>
                        <average_latency>0</average_latency>
                </fbc_stats>
                <ecc_mode>
                        <current_ecc>N/A</current_ecc>
                        <pending_ecc>N/A</pending_ecc>
                </ecc_mode>
                <ecc_errors>
                        <volatile>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </volatile>
                        <aggregate>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </aggregate>
                </ecc_errors>
                <retired_pages>
                        <multiple_single_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </multiple_single_bit_retirement>
                        <double_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </double_bit_retirement>
                        <pending_blacklist>N/A</pending_blacklist>
                        <pending_retirement>N/A</pending_retirement>
                </retired_pages>
                <remapped_rows>N/A</remapped_rows>
                <temperature>
                        <gpu_temp>60 C</gpu_temp>
                        <gpu_temp_tlimit>N/A</gpu_temp_tlimit>
                        <gpu_temp_max_threshold>104 C</gpu_temp_max_threshold>
                        <gpu_temp_slow_threshold>101 C</gpu_temp_slow_threshold>
                        <gpu_temp_max_gpu_threshold>N/A</gpu_temp_max_gpu_threshold>
                        <gpu_target_temperature>83 C</gpu_target_temperature>
                        <memory_temp>N/A</memory_temp>
                        <gpu_temp_max_mem_threshold>N/A</gpu_temp_max_mem_threshold>
                </temperature>
                <supported_gpu_target_temp>
                        <gpu_target_temp_min>65 C</gpu_target_temp_min>
                        <gpu_target_temp_max>98 C</gpu_target_temp_max>
                </supported_gpu_target_temp>
                <gpu_power_readings>
                        <power_state>P0</power_state>
                        <power_draw>30.08 W</power_draw>
                        <current_power_limit>75.00 W</current_power_limit>
                        <requested_power_limit>75.00 W</requested_power_limit>
                        <default_power_limit>75.00 W</default_power_limit>
                        <min_power_limit>75.00 W</min_power_limit>
                        <max_power_limit>75.00 W</max_power_limit>
                </gpu_power_readings>
                <gpu_memory_power_readings>
                        <power_draw>N/A</power_draw>
                </gpu_memory_power_readings>
                <module_power_readings>
                        <power_state>P0</power_state>
                        <power_draw>N/A</power_draw>
                        <current_power_limit>N/A</current_power_limit>
                        <requested_power_limit>N/A</requested_power_limit>
                        <default_power_limit>N/A</default_power_limit>
                        <min_power_limit>N/A</min_power_limit>
                        <max_power_limit>N/A</max_power_limit>
                </module_power_readings>
                <clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                        <sm_clock>1721 MHz</sm_clock>
                        <mem_clock>3499 MHz</mem_clock>
                        <video_clock>1544 MHz</video_clock>
                </clocks>
                <applications_clocks>
                        <graphics_clock>1075 MHz</graphics_clock>
                        <mem_clock>3504 MHz</mem_clock>
                </applications_clocks>
                <default_applications_clocks>
                        <graphics_clock>1075 MHz</graphics_clock>
                        <mem_clock>3504 MHz</mem_clock>
                </default_applications_clocks>
                <deferred_clocks>
                        <mem_clock>N/A</mem_clock>
                </deferred_clocks>
                <max_clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                        <sm_clock>1721 MHz</sm_clock>
                        <mem_clock>3504 MHz</mem_clock>
                        <video_clock>1556 MHz</video_clock>
                </max_clocks>
                <max_customer_boost_clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                </max_customer_boost_clocks>
                <clock_policy>
                        <auto_boost>N/A</auto_boost>
                        <auto_boost_default>N/A</auto_boost_default>
                </clock_policy>
                <voltage>
                        <graphics_volt>N/A</graphics_volt>
                </voltage>
                <fabric>
                        <state>N/A</state>
                        <status>N/A</status>
                        <cliqueId>N/A</cliqueId>
                        <clusterUuid>N/A</clusterUuid>
                        <health>
                                <bandwidth>N/A</bandwidth>
                        </health>
                </fabric>
                <supported_clocks>
                        <supported_mem_clock>
                                <value>3504 MHz</value>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>810 MHz</value>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>405 MHz</value>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                </supported_clocks>
                <processes>
                        <process_info>
                                <gpu_instance_id>N/A</gpu_instance_id>
                                <compute_instance_id>N/A</compute_instance_id>
                                <pid>1794890</pid>
                                <type>C</type>
                                <process_name></process_name>
                                <used_memory>203 MiB</used_memory>
                        </process_info>
                </processes>
                <accounted_processes>
                </accounted_processes>
        </gpu>

</nvidia_smi_log>

nvidia-smi -x -q on host:

<?xml version="1.0" ?>
<!DOCTYPE nvidia_smi_log SYSTEM "nvsmi_device_v12.dtd">
<nvidia_smi_log>
        <timestamp>Mon Mar 25 15:41:05 2024</timestamp>
        <driver_version>550.54.14</driver_version>
        <cuda_version>12.4</cuda_version>
        <attached_gpus>1</attached_gpus>
        <gpu id="00000000:00:10.0">
                <product_name>Quadro P2000</product_name>
                <product_brand>Quadro</product_brand>
                <product_architecture>Pascal</product_architecture>
                <display_mode>Disabled</display_mode>
                <display_active>Disabled</display_active>
                <persistence_mode>Disabled</persistence_mode>
                <addressing_mode>N/A</addressing_mode>
                <mig_mode>
                        <current_mig>N/A</current_mig>
                        <pending_mig>N/A</pending_mig>
                </mig_mode>
                <mig_devices>
                        None
                </mig_devices>
                <accounting_mode>Disabled</accounting_mode>
                <accounting_mode_buffer_size>4000</accounting_mode_buffer_size>
                <driver_model>
                        <current_dm>N/A</current_dm>
                        <pending_dm>N/A</pending_dm>
                </driver_model>
                <serial>0320218076820</serial>
                <uuid>GPU-68be6701-ee4c-5659-9384-45319d269070</uuid>
                <minor_number>0</minor_number>
                <vbios_version>86.06.3F.00.30</vbios_version>
                <multigpu_board>No</multigpu_board>
                <board_id>0x10</board_id>
                <board_part_number>900-5G410-0100-000</board_part_number>
                <gpu_part_number>1C30-875-A1</gpu_part_number>
                <gpu_fru_part_number>N/A</gpu_fru_part_number>
                <gpu_module_id>1</gpu_module_id>
                <inforom_version>
                        <img_version>G410.0502.00.02</img_version>
                        <oem_object>1.1</oem_object>
                        <ecc_object>N/A</ecc_object>
                        <pwr_object>N/A</pwr_object>
                </inforom_version>
                <inforom_bbx_flush>
                        <latest_timestamp>N/A</latest_timestamp>
                        <latest_duration>N/A</latest_duration>
                </inforom_bbx_flush>
                <gpu_operation_mode>
                        <current_gom>N/A</current_gom>
                        <pending_gom>N/A</pending_gom>
                </gpu_operation_mode>
                <c2c_mode>N/A</c2c_mode>
                <gpu_virtualization_mode>
                        <virtualization_mode>Pass-Through</virtualization_mode>
                        <host_vgpu_mode>N/A</host_vgpu_mode>
                        <vgpu_heterogeneous_mode>N/A</vgpu_heterogeneous_mode>
                </gpu_virtualization_mode>
                <gpu_reset_status>
                        <reset_required>No</reset_required>
                        <drain_and_reset_recommended>N/A</drain_and_reset_recommended>
                </gpu_reset_status>
                <gsp_firmware_version>N/A</gsp_firmware_version>
                <ibmnpu>
                        <relaxed_ordering_mode>N/A</relaxed_ordering_mode>
                </ibmnpu>
                <pci>
                        <pci_bus>00</pci_bus>
                        <pci_device>10</pci_device>
                        <pci_domain>0000</pci_domain>
                        <pci_base_class>3</pci_base_class>
                        <pci_sub_class>0</pci_sub_class>
                        <pci_device_id>1C3010DE</pci_device_id>
                        <pci_bus_id>00000000:00:10.0</pci_bus_id>
                        <pci_sub_system_id>11B31028</pci_sub_system_id>
                        <pci_gpu_link_info>
                                <pcie_gen>
                                        <max_link_gen>3</max_link_gen>
                                        <current_link_gen>3</current_link_gen>
                                        <device_current_link_gen>3</device_current_link_gen>
                                        <max_device_link_gen>3</max_device_link_gen>
                                        <max_host_link_gen>N/A</max_host_link_gen>
                                </pcie_gen>
                                <link_widths>
                                        <max_link_width>16x</max_link_width>
                                        <current_link_width>8x</current_link_width>
                                </link_widths>
                        </pci_gpu_link_info>
                        <pci_bridge_chip>
                                <bridge_chip_type>N/A</bridge_chip_type>
                                <bridge_chip_fw>N/A</bridge_chip_fw>
                        </pci_bridge_chip>
                        <replay_counter>0</replay_counter>
                        <replay_rollover_counter>0</replay_rollover_counter>
                        <tx_util>30 KB/s</tx_util>
                        <rx_util>31 KB/s</rx_util>
                        <atomic_caps_inbound>N/A</atomic_caps_inbound>
                        <atomic_caps_outbound>N/A</atomic_caps_outbound>
                </pci>
                <fan_speed>57 %</fan_speed>
                <performance_state>P0</performance_state>
                <clocks_event_reasons>
                        <clocks_event_reason_gpu_idle>Not Active</clocks_event_reason_gpu_idle>
                        <clocks_event_reason_applications_clocks_setting>Not Active</clocks_event_reason_applications_clocks_setting>
                        <clocks_event_reason_sw_power_cap>Not Active</clocks_event_reason_sw_power_cap>
                        <clocks_event_reason_hw_slowdown>Not Active</clocks_event_reason_hw_slowdown>
                        <clocks_event_reason_hw_thermal_slowdown>Not Active</clocks_event_reason_hw_thermal_slowdown>
                        <clocks_event_reason_hw_power_brake_slowdown>Not Active</clocks_event_reason_hw_power_brake_slowdown>
                        <clocks_event_reason_sync_boost>Not Active</clocks_event_reason_sync_boost>
                        <clocks_event_reason_sw_thermal_slowdown>Not Active</clocks_event_reason_sw_thermal_slowdown>
                        <clocks_event_reason_display_clocks_setting>Not Active</clocks_event_reason_display_clocks_setting>
                </clocks_event_reasons>
                <sparse_operation_mode>N/A</sparse_operation_mode>
                <fb_memory_usage>
                        <total>5120 MiB</total>
                        <reserved>73 MiB</reserved>
                        <used>205 MiB</used>
                        <free>4840 MiB</free>
                </fb_memory_usage>
                <bar1_memory_usage>
                        <total>256 MiB</total>
                        <used>2 MiB</used>
                        <free>254 MiB</free>
                </bar1_memory_usage>
                <cc_protected_memory_usage>
                        <total>0 MiB</total>
                        <used>0 MiB</used>
                        <free>0 MiB</free>
                </cc_protected_memory_usage>
                <compute_mode>Default</compute_mode>
                <utilization>
                        <gpu_util>11 %</gpu_util>
                        <memory_util>15 %</memory_util>
                        <encoder_util>58 %</encoder_util>
                        <decoder_util>55 %</decoder_util>
                        <jpeg_util>N/A</jpeg_util>
                        <ofa_util>N/A</ofa_util>
                </utilization>
                <encoder_stats>
                        <session_count>1</session_count>
                        <average_fps>445</average_fps>
                        <average_latency>2464</average_latency>
                </encoder_stats>
                <fbc_stats>
                        <session_count>0</session_count>
                        <average_fps>0</average_fps>
                        <average_latency>0</average_latency>
                </fbc_stats>
                <ecc_mode>
                        <current_ecc>N/A</current_ecc>
                        <pending_ecc>N/A</pending_ecc>
                </ecc_mode>
                <ecc_errors>
                        <volatile>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </volatile>
                        <aggregate>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </aggregate>
                </ecc_errors>
                <retired_pages>
                        <multiple_single_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </multiple_single_bit_retirement>
                        <double_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </double_bit_retirement>
                        <pending_blacklist>N/A</pending_blacklist>
                        <pending_retirement>N/A</pending_retirement>
                </retired_pages>
                <remapped_rows>N/A</remapped_rows>
                <temperature>
                        <gpu_temp>57 C</gpu_temp>
                        <gpu_temp_tlimit>N/A</gpu_temp_tlimit>
                        <gpu_temp_max_threshold>104 C</gpu_temp_max_threshold>
                        <gpu_temp_slow_threshold>101 C</gpu_temp_slow_threshold>
                        <gpu_temp_max_gpu_threshold>N/A</gpu_temp_max_gpu_threshold>
                        <gpu_target_temperature>83 C</gpu_target_temperature>
                        <memory_temp>N/A</memory_temp>
                        <gpu_temp_max_mem_threshold>N/A</gpu_temp_max_mem_threshold>
                </temperature>
                <supported_gpu_target_temp>
                        <gpu_target_temp_min>65 C</gpu_target_temp_min>
                        <gpu_target_temp_max>98 C</gpu_target_temp_max>
                </supported_gpu_target_temp>
                <gpu_power_readings>
                        <power_state>P0</power_state>
                        <power_draw>31.94 W</power_draw>
                        <current_power_limit>75.00 W</current_power_limit>
                        <requested_power_limit>75.00 W</requested_power_limit>
                        <default_power_limit>75.00 W</default_power_limit>
                        <min_power_limit>75.00 W</min_power_limit>
                        <max_power_limit>75.00 W</max_power_limit>
                </gpu_power_readings>
                <gpu_memory_power_readings>
                        <power_draw>N/A</power_draw>
                </gpu_memory_power_readings>
                <module_power_readings>
                        <power_state>P0</power_state>
                        <power_draw>N/A</power_draw>
                        <current_power_limit>N/A</current_power_limit>
                        <requested_power_limit>N/A</requested_power_limit>
                        <default_power_limit>N/A</default_power_limit>
                        <min_power_limit>N/A</min_power_limit>
                        <max_power_limit>N/A</max_power_limit>
                </module_power_readings>
                <clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                        <sm_clock>1721 MHz</sm_clock>
                        <mem_clock>3499 MHz</mem_clock>
                        <video_clock>1544 MHz</video_clock>
                </clocks>
                <applications_clocks>
                        <graphics_clock>1075 MHz</graphics_clock>
                        <mem_clock>3504 MHz</mem_clock>
                </applications_clocks>
                <default_applications_clocks>
                        <graphics_clock>1075 MHz</graphics_clock>
                        <mem_clock>3504 MHz</mem_clock>
                </default_applications_clocks>
                <deferred_clocks>
                        <mem_clock>N/A</mem_clock>
                </deferred_clocks>
                <max_clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                        <sm_clock>1721 MHz</sm_clock>
                        <mem_clock>3504 MHz</mem_clock>
                        <video_clock>1556 MHz</video_clock>
                </max_clocks>
                <max_customer_boost_clocks>
                        <graphics_clock>1721 MHz</graphics_clock>
                </max_customer_boost_clocks>
                <clock_policy>
                        <auto_boost>N/A</auto_boost>
                        <auto_boost_default>N/A</auto_boost_default>
                </clock_policy>
                <voltage>
                        <graphics_volt>N/A</graphics_volt>
                </voltage>
                <fabric>
                        <state>N/A</state>
                        <status>N/A</status>
                        <cliqueId>N/A</cliqueId>
                        <clusterUuid>N/A</clusterUuid>
                        <health>
                                <bandwidth>N/A</bandwidth>
                        </health>
                </fabric>
                <supported_clocks>
                        <supported_mem_clock>
                                <value>3504 MHz</value>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>810 MHz</value>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>405 MHz</value>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                </supported_clocks>
                <processes>
                        <process_info>
                                <gpu_instance_id>N/A</gpu_instance_id>
                                <compute_instance_id>N/A</compute_instance_id>
                                <pid>1794890</pid>
                                <type>C</type>
                                <process_name>/usr/lib/jellyfin-ffmpeg/ffmpeg</process_name>
                                <used_memory>203 MiB</used_memory>
                        </process_info>
                </processes>
                <accounted_processes>
                </accounted_processes>
        </gpu>

</nvidia_smi_log>
EuleMitKeule commented 8 months ago

metrics.out:

nvidia_smi,arch=Pascal,compute_mode=Default,host=docker,index=0,name=Quadro\ P2000,pstate=P0,uuid=GPU-68be6701-ee4c-5659-9384-45319d269070 memory_free=4840i,memory_reserved=73i,encoder_stats_session_count=1i,fbc_stats_session_count=0i,clocks_current_sm=1721i,vbios_versio>
nvidia_smi_process,host=docker,type=C pid=1794890i,used_memory=203i 1711377820000000000
EuleMitKeule commented 8 months ago

Apparently it's not a problem with Telegraf, nvidia-smi just does not display the process name when executed in a container.

EuleMitKeule commented 8 months ago

I fixed it by adding pid: host to the docker-compose configuration. Thanks for the help!