All installed systemd units are forcefully enabled on first boot

alex3d commented 7 months ago

What happened?

All systemd units are forcefully enabled on first boot, including those that does not enabled by default on package install (rsync and iperf3 services, for example) or even those that specifically disabled during armbian build process (smartmontools): https://github.com/armbian/build/blob/9c75c86a5488b043b76ba39305d0113d98885688/lib/functions/rootfs/distro-specific.sh#L13-L15

It seems systemd's firstboot concept is not an ideal match for debian/ubuntu. AFAIC systemd expects empty/unpopulated /etc on first boot and populates it from all units defined in /usr ignoring package manager defaults or services made disabled during debootstrap or image build. Systemd firstboot behavior triggered by https://github.com/armbian/build/blob/9c75c86a5488b043b76ba39305d0113d98885688/lib/functions/rootfs/rootfs-create.sh#L229-L231

An ideal solution would be to regenerate only /etc/machine-id file on first boot, not triggering full systemd's "firstboot" procedure. Unfortuntely I have not found solution except manual image file patching before first boot. Eg, mount SD image file, rm /mnt/sd/etc/machine-id, systemd-firstboot --root=/mnt/sd --setup-machine-id

Userspace release: bookworm

How to reproduce?

1) Disable iperf3 service:

# systemctl disable iperf3
# systemctl is-enabled iperf3
disabled

2) Reboot. As expected iperf3 is still disabled after reboot 3) Now, rm /etc/machine-id and then reboot once again. After that:

# systemctl is-enabled iperf3
enabled
# systemctl is-active iperf3
active

Branch

main (main development branch)

On which host OS are you observing this problem?

Other

Relevant log URL

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 7 months ago

Jira ticket: AR-2051

viraniac commented 7 months ago

Thanks for reporting this. I also am seeing this behavior on vim4/vim1s and was trying to troubleshoot the same. At the same time, I was not able to reproduce this on Allwinner boards which was weird. Which board did you experienced this on?

alex3d commented 7 months ago

rockchip64/rk3318-box

viraniac commented 7 months ago

@alex3d I found this in the bookworms documentation

4.If /etc/machine-id exists and is empty, a boot is not considered the first boot. systemd will still bind-mount a file containing the actual machine-id over it and later try to commit it to disk (if /etc/ is writable).

I am currently creating an image for vim1s to test if an empty /etc/machine-id makes the systemd behave same as it does on jammy.

viraniac commented 7 months ago

Also found

If no preset files exist, preset operations will enable all units that are installed by default. If this is not desired and all units shall rather be disabled, it is necessary to ship a preset file with a single, catchall "disable *" line.

Example 1. Default to off

# /lib/systemd/system-preset/99-default.preset
disable *

This disables all units. Due to the filename prefix "99-", it will be read last and hence can easily be overridden by spin or administrator preset policy.

viraniac commented 7 months ago

Comparing between https://manpages.debian.org/bullseye/systemd/machine-id.5.en.html and https://manpages.debian.org/bookworm/systemd/machine-id.5.en.html

There is a added step of preseting the units on first-boot. ~So I guess the right solution here will be to add a default preset to disable all units like mentioned in the previous comment~ On second thought I guess that will disable new units on package installation as well so probably not so good idea then.

alex3d commented 7 months ago

@viraniac I can confirm empty /etc/machine-id have fixed issue.

Btw, I've found "ignore" preset option, that could have been the best option (it opens possibility to use firstboot condition as unit start trigger, for partition resize, etc)

# /usr/lib/systemd/system-preset/99-default.preset
ignore *

But unfortunately this option is very very recent feature from systemd 254, so this is no go for a near future. bookworm have systemd 252, latest ubuntu noble will have systemd 253.

viraniac commented 7 months ago

Btw, I've found "ignore" preset option, that could have been the best option (it opens possibility to use firstboot condition as unit start trigger, for partition resize, etc)
# /usr/lib/systemd/system-preset/99-default.preset
ignore *
But unfortunately this option is very very recent feature from systemd 254, so this is no go for a near future. bookworm have systemd 252, latest ubuntu noble will have systemd 253.

I guess we have to wait to see if we can use that. When I tried "disable *" option I noticed that systemd didn't enabled all units which is what we expect. But at the same time installing packages like apache2 also didn't enabled its unit which user might expect as default behavior. I guess same also has to be tested for the ignore option. Systemd 255 is available on trixie and sid so its possible we can test it on them.

One option probably can also be to install presets with "disable " or "ignore " in them and remove the preset file on first boot

igorpecovnik commented 7 months ago

Should be o.k., re-open if not.

armbian / build