Open aburmash opened 6 months ago
coiby
commented
6 months ago Hi @aburmash,
Thanks for reporting this issue! I'm curious to ask why do you think that NetworkManager daemon doesn't get restarted after the update "is expected and normal as well"? What's the risk of auto-restart NM after updating NM?
aburmash
commented
6 months ago Potential network loss. LAN connections are likely to persist through NM restart, but some connections may be interrupted during restart for a while, which specifically is a problem for enterprise environment. Most ( if not all ) RH based distros follow RH logic and reload NM instead of restarting NM. Change in NM package itself goes back to https://bugzilla.redhat.com/show_bug.cgi?id=811200
I believe the main problem itself is not just the requirement to restart NM + kdump or reboot the machine, the problem is that when you do yum update, that pulls in NM with similar change AND also some package that triggers kdump initramfs rebuild, kdump service is triggered in background ( because of monitored files change ), not as a post-script, so when kdump service post/during yum update fails, you actually have no idea that kdump is not OK, unless you specifically decide to inspect journalctl / systemctl status kdump.
coiby
commented
6 months ago Thanks for explaining the side-effect of restarting NM to me!
I believe kdump.service gets started because kexec-tools (kdump-utils) itself gets updated and the following scriptlet is triggered
%postun
%systemd_postun_with_restart kdump.serviceCan you confirm it? If that's the case, maybe we can add an exception for this case where "Warning: nmcli (1.46.0) and NetworkManager (1.44.0) versions don't match. Restarting NetworkManager is advised" is detected.
When system with ISCSI network disk used for system partition ( / ) is upgraded, and there are packages in the dnf transaction that trigger kdump initramfs rebuild AND new NetworkManager is installed as an update in the same transaction, kdump service will fail and kdump initramfs will not be regenerated. This happens for example during update from NetworkManager 1.44 to 1.46 ( centos-stream case ).
This is caused by a combination of expected NetworkManager behaviour and kdump using nmcli ( NetworkManager client to manage connections ) to generate kdump initramfs image with network.
After update NetworkManager does NOT restart the daemon ( this is expected and normal as well ), but only reloads the configuration. That causes the mismatch between running daemon and installed libs and clients ( also expected ). Certain nmcli commands fail, because new NetworkManager option added in 1.46 is not supported by running 1.44 daemon ( also expected ).
For example this command fails: https://github.com/rhkdump/kdump-utils/blob/main/dracut-module-setup.sh#L262 Error emitted in kdump logs is something like:
This is caused by the fact that running NM daemon 1.44 has no idea about a NEW 1.46 option
autoconnect-ports( that was added in 1.46, but was missing in 1.44 )nmcli also emits a message suggesting restarting NetworkManager - "Warning: nmcli (1.46.0) and NetworkManager (1.44.0) versions don't match. Restarting NetworkManager is advised." This message is seen in journalctl logs and kdump service logs.
kdump running those nmcli commands will fail to do so and fail the service. During the update that triggers both updates of packages that trigger kdump rebuld and NM update user may end up with non-functional kdump up until reboot or NM + kdump service restart.
Not sure what should be the proper fix. From one point of view, "restart your system or restart NM + kdump" is a solution, but is there a better suggestion to look into ?