Repeat execution on the timer seems to be inconsistent

[brianread108]

Describe the bug Watching the logs I am not seeing the LXC autoscale system re-running every 600 seconds after a change has been made. However if I then do a restart and the run does not make any changes, then it then seems to re-run every 600 seconds after that. I have seen this a number of times.

To Reproduce

Watch the logs

Expected behavior

It runs every 600 seconds irrespective of if a change has been made.

Screenshots none

Desktop (please complete the following information): Mint 22

Smartphone (please complete the following information): n/a Additional context

Server: CPU(s) 8 x Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz (1 Socket) Kernel Version Linux 6.8.12-2-pve (2024-09-05T10:03Z) Boot Mode Legacy BIOS Manager Version pve-manager/8.2.7/3e0176e6bb2ade3b

[brianread108]

Just spotted that when NO change is made there is a final log message:

Resource allocation process completed. Next run in 600 seconds.

The message does not appear if a change is made.

[brianread108]

Here's the log where the problem shows up:

2024-11-09 09:03:16 - INFO - Starting resource allocation process... 2024-11-09 09:03:16 - INFO - Ignoring LXC Containers: set() 2024-11-09 09:03:16 - INFO - Initial resources before adjustments: 6 cores, 27945 MB memory 2024-11-09 09:03:16 - INFO - Current resource usage for all containers: 2024-11-09 09:03:16 - INFO - Container 104: CPU usage: 0.0%, Memory usage: 38.03MB (92.57% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 109: CPU usage: 0.0%, Memory usage: 9.88MB (98.07% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 100: CPU usage: 2.0%, Memory usage: 12.49MB (97.56% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 105: CPU usage: 0.0%, Memory usage: 15.17MB (97.04% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 114: CPU usage: 0.0%, Memory usage: 15.67MB (96.94% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 117: CPU usage: 0.0%, Memory usage: 7.94MB (98.45% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 119: CPU usage: 1.0%, Memory usage: 73.32MB (97.14% free of 2560MB total), Initial cores: 2, Initial memory: 2560MB 2024-11-09 09:03:16 - INFO - Container 120: CPU usage: 0.0%, Memory usage: 44.4MB (91.33% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 136: CPU usage: 6.0%, Memory usage: 61.08MB (94.04% free of 1024MB total), Initial cores: 1, Initial memory: 1024MB 2024-11-09 09:03:16 - INFO - Container 137: CPU usage: 0.0%, Memory usage: 45.41MB (96.12% free of 1169MB total), Initial cores: 1, Initial memory: 1169MB 2024-11-09 09:03:16 - INFO - Container 131: CPU usage: 37.0%, Memory usage: 56.78MB (96.3% free of 1536MB total), Initial cores: 1, Initial memory: 1536MB 2024-11-09 09:03:16 - INFO - Container 126: CPU usage: 0.0%, Memory usage: 66.32MB (87.05% free of 512MB total), Initial cores: 1, Initial memory: 512MB 2024-11-09 09:03:16 - INFO - Container 119 - CPU usage below lower threshold. 2024-11-09 09:03:16 - INFO - Container 119 - Decrement: 1, New cores: 1 2024-11-09 09:57:36 - INFO - Starting resource allocation process... 2

(Sorry can't work out why that log is ignoring newlines - it was copy and pasted from terminal window)

[fabriziosalmi]

I never encounter such issue but once I have some time ofc I will try to investigate, if confirmed should be quite easy to fix. TY to point me out to that.

[brianread108]

Here's a log of when it fails to setup the next timer event (with DEBUG set), a log of the status of all the lxc systemd daemons at that failure point, and also a DEBUG log of when it works (when no changes are made). As you can see it looks as though the loadavg is failing (I do have the -l set). But I do not immediatly see how that effects things. proxmox-lxcAutoscalingErrors.zip

Hope this helps.

[brianread108]

Incidentally I tried the pct set 131 -cores 2 command at the console, and it worked silently as you'd expect.