search

gpuopenanalytics / pynvml

Provide Python access to the NVML library for GPU diagnostics
BSD 3-Clause "New" or "Revised" License
205 stars 31 forks source link

NVML Permissions #19

Closed quasiben closed 4 years ago

quasiben commented 4 years ago

I'm filing this issue not for pynvml to resolve , but rather to raise awareness for permissioning issues around nvml to the pynvml devs.

NVML can be administered in such away as to prevent non-admin/root users access to the library. This would prevent nvml from querying performance counters. For example:

(base) bzaitlen@dgx15:~$ nvidia-smi nvlink -g 0
GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-96ab329d-7a1f-73a8-a9b7-18b4b2855f92)
NVML: Unable to get the NvLink link utilization counter control for link 0: Insufficient Permissions

I found this page on admin pemissioning which explains how to set/unset admin privileges.

Another way to easily check the status of permissions is to look for RmProfilingAdminOnly in driver params:

(base) bzaitlen@dgx15:~$ cat /proc/driver/nvidia/params | grep RmProfilingAdminOnly
RmProfilingAdminOnly: 1
rjzamora commented 4 years ago

Thaks @quasiben! I do think it would be good for us to document these permissioning issues in pynvml - Perhaps some of the information you are sharing can be mentioned in the README

rjzamora commented 4 years ago

Thanks again @quasiben! I merged the doc changes in #20 -- Feel free to suggest other NVML-permissioning action items on the pynvml side.