`xen-balloon` consumes 15-30%+ CPU on idle in user qubes

[jamke]

Qubes OS release

R4.1 Used template is based on fedora-38-minimal. Both OS and template are up to date.

Brief summary

xen-balloon command consumes 30%+ CPU on idle. Happens with several qubes more often that others based on the same template.

Steps to reproduce

Use qube A while qube B shows no applications and should idle and do nothing.

Expected behavior

Qube B consumes 0% CPU

Actual behavior

Qube B consumes 15-35% CPU constantly or from time to time. top shows that xen-balloon command is the only one that consumes this.

Additional

• This problem of xen-balloon consuming CPU already happened in past but for dom0 (in 2021-2022): https://forum.qubes-os.org/t/high-dom0-cpu-usage-by-xen-balloon/6732/ But I have everything up to date for R4.1, so it is expected to be fixed long time ago. i also have it in AppVMs, not dom0. Probably related, though.

• It's probably a regression of the last year or so, before that I have never had this situation before, even with the same template.

[UndeadDevel]

I've seen this symptom (~15-20% CPU usage in idle qubes) as well quite often (not sure about 4.1, but definitely in 4.2), but every time I launch a terminal in them and execute top, the CPU usage drops to very low (normal) levels again...did you use an admin console from a management disp VM to "catch" the xen-balloon process? The one thing that sometimes flashed for a moment with high-ish CPU usage in top was the Xorg process, but that may just be related to the fact that I just launched a terminal in that qube.

Edit: I've now been able to confirm that it's not xen-balloon, but Xorg and at-spi2 who are the culprits.

[jamke]

@UndeadDevel I use terminal (konsole), and top. Nothing fancy in my case. xen-balloon is shown as the top consumer. Can go to 0% in a few seconds, or can stay 15-35% for a longer time, for like minutes.

[neowutran]

I had the same issue some months ago. Multiple VM with memory balancing enabled, max memory of all those VM combined exceeded the machine physical RAM. And it feel like they were competing to grab as much ram as possible while not actually needing it.

I ended up disabling memory balancing for VMs that don't have a lot of RAM consumption variation

[marmarek]

And it feel like they were competing to grab as much ram as possible while not actually needing it.

This is by design - qmemman assign as much memory as possible (up to maxmem). Linux will use excess memory for disk cache, which does help with performance.

[jamke]

@marmarek what is xen-balloon used for? Search does not show much for me, it's not even in where results.

[marmarek]

xen-balloon is a kernel thread responsible for dynamically resizing VM memory