needrestart: restarts cloud-final.service on apt dist-upgrade resulting in SIGTERM which prevents APT packages from being installed.

[blackboxsw]

Bug report

When #cloud-config userdata contains package_upgrade: true on debian-based environments, cloud-final.service runs apt-get dist-upgrade. On Ubuntu Noble needrestart may get involved and indicate that cloud-final.service is one of those services needing a restart.

A forced restart of cloud-final.service will send a SIGTERM to the oneshot cloud-final.service and avoid to remaining configuration operations such as additional APT package installs on the system leaving cloud-init in a state where it hasn't fully performed all system configuration.

Tracebacks can be seen in /var/log/cloud-init.log representing this condition and cloud-init-output.log will represent that needrestart targets cloud-final.service.

Steps to reproduce the problem

cat needrestart.yaml <<EOF
#cloud-config
# failure case in ubuntu-noble where needrestart /etc/cloud/build.info serial: 20240312
package_update: true
package_upgrade: true
packages: [sl]
EOF

lxc launch ubuntu-daily:noble nn-test -c cloud-init.user-data="$(cat needrestart.yaml)"
lxc exec nn-test -- cloud-init status --wait --format=yaml # expect no errors/warnings

# confirm SIGTERM just after apt-get dist-upgrade
$ lxc exec nn-test -- grep -B 1 SIGTERM /var/log/cloud-init.log
2024-03-27 16:55:34,121 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'dist-upgrade'] with allowed return codes [0] (shell=False, capture=False)
2024-03-27 16:55:46,351 - util.py[DEBUG]: Cloud-init 24.1-0ubuntu1 received SIGTERM, exiting...

# confirm needrestart is the cause of that issue
$ lxc exec nn-test -- grep 'Restarting services' -A 1  /var/log/cloud-init-output.log 
Restarting services...
 systemctl restart cloud-final.service console-getty.service cron.service packagekit.service polkit.service systemd-hostnamed.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-timedated.service systemd-udevd.service udisks2.service

# confirm no deb package sl installed
$ lxc exec nn-test -- dpkg -l sl
dpkg-query: no packages found matching sl

Environment details

• Cloud-init version: 24.1.3
• Operating System Distribution: Ubuntu Noble
• Cloud provider, platform or installer type: lxd_container

cloud-init logs

Provided above

[blackboxsw]

Downstream MR put up for Ubuntu https://bugs.launchpad.net/ubuntu/+source/needrestart/+bug/2059337 for discussion on this.

[blackboxsw]

Downstream MR released in needrestart deb package version 3.6-7ubuntu4 for Ubuntu. This is not an issue in cloud-init specifically.

[holmanb]

@blackboxsw thanks for working through this and getting this fixed in needsrestart!