How to detect iGPU free memory

[zhewang1-intc]

Hi, I'd like to ask how to use level-zero/sycl to detect available memory of Intel iGPUs (e.g., Intel® Iris® Xe Graphics, PCI ID: 46A0, codename: Alder Lake-P).

I found relevant APIs in level-zero, such as zesDeviceEnumMemoryModules and zesMemoryGetState. Unfortunately, when I call zesDeviceEnumMemoryModules on a system with only iGPU, the return value of pCount is 0. Therefore, I cannot get the iGPU memory information through zesMemoryGetState.

i also found how to use sycl to detect free memory in this doc, here is the code

if (dev.has(aspect::ext_intel_free_memory)) {
  auto FreeMemory = dev.get_info<ext::intel::info::device::free_memory>();
}

I ran this func with set ZES_ENABLE_SYSMAN=1, but I got the free_mem size is 0.

note that these issues don't appear on Intel dGPUs. Windows Task Manager can directly show the total and used memory of iGPU. How to implement this function? Are there any code examples?

[saik-intel]

hi @zhewang1-intc Level zero Sysman doesn't support most of the modules for IGPU case. Memory module majorly designed for DGpus only, hence you are getting 0 pCount that means no handle will be created hence you can't query on iGPU.

[eero-t]

iGPUs do not have their own (device) memory, they share system memory with CPU.

While there are many ways to get amount of system memory, GPU drivers do not allow allocating all system memory to GPU for obvious reasons (nothing would work unless quite a bit of memory is left for CPU). These limits may change between kernel and user-space driver versions. Best is to query specific (media, 3D, compute) user-space driver for amount of "free" GPU memory using its normal APIs.

PS. There's similar thing e.g. with power, iGPU is on the "uncore" part of the CPU package with LLC, which it shares with CPU. And you query "uncore" power usage with RAPL similarly to CPU power usage.

[zhewang1-intc]

Thanks @eero-t for your feedback, could you pls tell me where I can find these normal APIs?

[eero-t]

Hm. On quick googling, it appears that OpenGL does not have a standard one (like I thought). You can query memory through OpenCL though: https://stackoverflow.com/questions/3568115/how-do-i-determine-available-device-memory-in-opencl

As to the GPU system memory usage, you can get that from i915 kernel driver debugfs entry:

$ sudo sed 's/^.*, \([0-9]\+\) bytes$/\1/' /sys/kernel/debug/dri/0/i915_gem_objects |\
  awk '{print $1/(1024*1024), "MiB"; exit}'

However, debugfs is unstable kernel API, access to it requires root, and only iGPUs use system memory, so above is not really relevant for dGPUs.

(There's a bit of system memory usage for dGPUs too, but unless device is out of local RAM i.e. paging it constantly to system memory and back, it's not significant.)

[zhewang1-intc]

@eero-t thanks for your quick reply, can we get the memory usage by a binary/lib(can implement it by ourself) without root privilege on Linux?

now I'm trying to add intel iGPU support for a popular LLM-related project named ollama in Nvidia/AMD GPU discovery process, it will check free GPU memory and then calculate how many layers can be offloaded to GPU, so i'd like to follow this process and I don't think users will run ollama application with sudo privilege on linux.

BTW I found on windows platform there is a simple script can get what I needed without Administrator privilege in here: https://stackoverflow.com/questions/73697470/is-there-a-way-to-get-intels-integrated-gpu-usage-using-python-on-windows now just lack Linux solution.

[eero-t]

@eero-t thanks for your quick reply, can we get the memory usage by a binary/lib(can implement it by ourself) without root privilege on Linux?

No admin is going to mount debugfs as non-root as that would make escalating local exploits trivial.

Another possibility besides debugfs and L0 Sysman APIs would be querying kernel directly using IOCTL: https://docs.kernel.org/gpu/driver-uapi.html#c.drm_i915_gem_memory_class

But on quick read I didn't grok whether that would provide any usable values for system memory on iGPUs. Also, if it would, then it would make sense to have L0 Sysman backend to provide system memory info in addition to device local memory info.

That's not currently the case, although spec has define for system memory "location": https://spec.oneapi.io/level-zero/latest/sysman/api.html#_CPPv420zes_mem_properties_t

now I'm trying to add intel iGPU support for a popular LLM-related project named ollama in Nvidia/AMD GPU discovery process, it will check free GPU memory and then calculate how many layers can be offloaded to GPU, so i'd like to follow this process and I don't think users will run ollama application with sudo privilege on linux.

As a quick workaround, what about just adding parameter for GPU memory usage?

(If there's not enough of whatever GPU memory ollama app allocates, the program can just exit with "not enough memory available" error.)

BTW I found on windows platform there is a simple script can get what I needed without Administrator privilege in here: https://stackoverflow.com/questions/73697470/is-there-a-way-to-get-intels-integrated-gpu-usage-using-python-on-windows now just lack Linux solution.

Does it actually work (without admin privilege)?

[zhewang1-intc]

Does it actually work (without admin privilege)?

i just run this code snippet without admin privilege in powershell, it works (((Get-Counter "\GPU Process Memory(*)\Local Usage").CounterSamples | where CookedValue).CookedValue | measure -sum).sum

As a quick workaround, what about just adding parameter for GPU memory usage?

make sense, will we support detecting free iGPU memory in the future? I think it's quite important feature as our iGPUs are more and more powerful, many users will use iGPU to do some inference tasks directly which may need to detect free vram before inference

[eero-t]

As a quick workaround, what about just adding parameter for GPU memory usage?

make sense, will we support detecting free iGPU memory in the future?

It may make sense to add more specificly titled bug about that: "Sysman returns only GPU device memory info, GPU system memory info missing". System memory is part of L0 Sysman spec, so info about that should IMHO also be returned.

[saik-intel]

@eero-t Currently L0 Sysman driver is written only for GPUs and i don't think we have any near future plan to support the System memory for IGPU use case.

[eero-t]

@eero-t Currently L0 Sysman driver is written only for GPUs and i don't think we have any near future plan to support the System memory for IGPU use case.

@saik-intel System memory is relevant both for dGPU and iGPU. It's true that in normal situations there's only little system memory used with dGPUs, but if dGPU starts running out of device memory, it may start using also significant amount of system memory.

(For now that's fairly extreme case for dGPUs due to paging slowdown, but if GPUs ever get cgroup support, it can be more common.)

[saik-intel]

@eero-t I agree L0 Spec says about system memory query via https://spec.oneapi.io/level-zero/latest/sysman/api.html#_CPPv4N13zes_mem_loc_t18ZES_MEM_LOC_SYSTEME handle but currently we return NULL handle for System memory query , as of now there is no such demand from customer to support.

[eero-t]

@zhewang1-intc Are there other use-cases and/or AI programs where getting used + available iGPU (system) memory amounts would be useful?

FYI: this was raised internally. If kernel turns out to already have reliable reporting of system memory used by GPU, supporting it in the L0 Sysman backend could be considered => list of use-cases where it would be useful, would be helpful for that consideration.