pynvml.nvml.NVMLError: System is not in ready state

[ainhoaVivel]

• Python version: 3.9.19
• Operating System: AlmaLinux 9.3 (Shamrock Pampas Cat)
• Pynvml version: 11.5.0

Description

I am using CodeCarbon to make some consumption measurements. However, this library uses pynvml in the background to access the graph information. I asked in the project repository and it seems that my problem is that pynvml is not working properly.

What I Did

I created this script

import pynvml

try:
    pynvml.nvmlInit()
    print("NVML initialized successfully")

    handle = pynvml.nvmlDeviceGetHandleByIndex(0)
    print(f"Device 0: {pynvml.nvmlDeviceGetName(handle)}")

    total_energy = pynvml.nvmlDeviceGetTotalEnergyConsumption(handle)
    print(f"Total energy consumption: {total_energy} mJ")

except pynvml.NVMLError as error:
    print(f"Failed to initialize NVML: {error}")

finally:
    pynvml.nvmlShutdown()

However, I got this output

NVML initialized successfully
Device 0: NVIDIA H100 PCIe
Failed to initialize NVML: System is not in ready state

I have tried several versions pynvml, but nothing. I can't find any additional information about the System is not in ready state error either. How can I fix this error?

[Lucas-Otavio]

I am also using CodeCarbon and I'm facing some similar issues, but the execution environment and error message are different.

I am trying to dockerize a project that uses Code Carbon, and it does not work inside the docker, even though nvidia-smi outputs as usual.

Environment:

• Container's Base: nvidia/cuda:12.5.0-devel-ubuntu22.04
• Python Version: 3.10.12
• pynvml version: 11.5.0
• Operating System: Ubuntu 22.04

Output: When running the same script, the error was different.

NVML initialized successfully
Device 0: NVIDIA GeForce GTX 980M
Failed to initialize NVML: Not Supported