Reboot required status determination has excessive impacts on CentOS

guillomovitch commented 7 months ago

Problem description

Since Version 2024.01, Uyuni is now able to properly determine reboot required status on non-SUSE distributions, which is great.

However, it turns out this determination has undesirable side-effects, at least on CentOS distribution, as it relies on running 'needs-restarting', or 'dnf -q needs-restarting -r' depending on version, every 10 seconds. The immediate consequence is having Salt minion logs, and DNF logs overfilled with endless repeated information. And I didn't checked yet impact on other resources (CPU, memory, netowrk...).

I personnaly don't think checking every 10s is really needed, whereas just checking once after each package update would be really enough. Unfortunatly, I didn"t found any way to modify this default frequency.

Steps to reproduce

Update Uyuni to version 2024.01 Update salt-minion version to latest available (3006.0-24.11.uyuni on CentOS9, 3006.0-25.1.uyuni on CentOS 8 and CentOS 7)

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2024.02-230900.213.1.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : out-of-date (version 2023.09-230900.207.8.uyuni3 installed)
Source package : Uyuni-Server-release-2024.02-230900.213.1.uyuni3.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

Typical extract from salt minion logs:
<code>
2024-03-14 11:54:52,489 [salt.loader.xxx.int.module.cmdmod:926 ][ERROR   ][15293] retcode: 1
2024-03-14 11:55:02,497 [salt.loader.xxx.int.module.cmdmod:920 ][ERROR   ][15293] Command 'needs-restarting' failed with return code: 1
2024-03-14 11:55:02,497 [salt.loader.xxxq.int.module.cmdmod:922 ][ERROR   ][15293] stdout: Core libraries or services have been updated:
  linux-firmware -> 20200421-82.git78c0348.el7_9

Reboot is required to ensure that your system benefits from these updates.

More information:
https://access.redhat.com/solutions/27943
</code>

Typical exctracts from DNF logs (default debug level is insane):
<code>
2024-03-14T10:32:25+0100 INFO --- logging initialized ---
2024-03-14T10:32:25+0100 DEBUG Loaded plugins: <invalid>, builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-man
ager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, susemanager, system-upgrade
2024-03-14T10:32:25+0100 DEBUG DNF version: 4.7.0
2024-03-14T10:32:25+0100 DEBUG cachedir: /var/cache/dnf
2024-03-14T10:32:25+0100 DEBUG Unknown configuration option: susemanager_token = eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE3Mjc4Njk4NjcsImlhdCI6MTY5NjMzMzg2NywibmJmIjoxNjk2MzMzNzQ3LCJq
dGkiOiJRTHRBWVJwaDNSam5xeC1HaEVBSDRBIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zOC14ODZfNjQiXX0.tyDKVVFfMztJasRtRM8h-lJkb5laFpZeCsQSVOUUQn8 in /etc/yum.repos.d/susemanager:ch
annels.repo
...
2024-03-14T10:32:25+0100 DEBUG Susemanager Plugin: [susemanager:centos8-x86_64] set token header: 'X-Mgr-Auth: ...sQSVOUUQn8'
...
2024-03-14T10:32:25+0100 DEBUG User-Agent: constructed: 'libdnf (CentOS Stream 8; generic; Linux.x86_64)'
</code>

Additional information

No response

rjmateus commented 7 months ago

Thank you for your report. This issue should be solved by this commit: https://github.com/uyuni-project/uyuni/pull/8383 It's planned to be part of the next uyuni release.

rjmateus commented 7 months ago

@wweellddeerr I think the PR will not address this concern. Any comment on it? thank you

guillomovitch commented 6 months ago

The PR will only fix salt minion logs issue, not dnf logs issue, nor abusive resource usage.

A real fix would be to stop running this query every 10s, as if real-time information was required. Actualising packages list seems to be done either through a regular server-scheduled task, either immediatly after package upgrade, why not use the same principle here ?

wweellddeerr commented 6 months ago

In other distributions, the verification is as simple as checking if a given file exists or not, so running it every 10 seconds doesn't seem to be an issue. However, considering that for CentOS the verification is not as straightforward, I think we should consider increasing the verification interval to every 30 seconds or perhaps every 1 minute.

In the package list refresh, we also update the required reboot data. However, initially, we decided it would be better to constantly check if a reboot is needed directly in the system to handle any situation. You can modify the verification interval or disable it entirely by creating a custom configuration file in the system, for instance, /etc/venv-salt-minion/minion.d/custom.conf:

beacons:
  reboot_info:
    interval: 120
    disabled: true

If you disable the beacon, the information will be updated with every package profile update. Therefore, I believe the behavior will align with what you just described.

guillomovitch commented 6 months ago

Indeed, this solution works. Thanks for the hint :)

avshiliaev commented 6 months ago

@wweellddeerr do you think there's anything from our side to fix/implement?

wweellddeerr commented 5 months ago

@wweellddeerr do you think there's anything from our side to fix/implement?

@avshiliaev there is an ongoing discussion regarding the best way to resolve this problem definitively. For now, there is a PR under review increasing the beacon execution interval.

uyuni-project / uyuni