Roadmap: systemd-sysext and systemd-confext

We have started to point users at systemd-sysext for bringing their software and configuration to Flatcar as overlay on /usr. To get there, a few workaround were needed. While there were discussions with upstream how to solve this, the work is yet to be done. The first step would be to drop our customizations through some upstream and Flatcar improvements. This should already allow us to transition to sysext as torcx replacement and for the OEM tools. Then we can look into unlocking more use cases for systemd-sysext and systemd-syscfg after doing the needed upstream development. Finally we can look at some syscfg use cases in Flatcar.

Drop our customizations to reach a state where there is one consistent future-proof sysext behavior supported by upstream:

[x] Make upstream systemd-sysext reload the units by default: systemd-sysext refresh should also issue a daemon-reload as done by systemctl enable X, this allows us to drop this line https://github.com/flatcar/init/blob/fbb1ec8e9d95598d65e4eac408dbc5f75b9b6138/systemd/system/ensure-sysext.service#L14: https://github.com/flatcar/Flatcar/issues/984
[x] https://github.com/flatcar/Flatcar/issues/766 which allows us to drop https://github.com/flatcar/init/blob/flatcar-master/systemd/system/systemd-sysext.service.d/10-mount-oem-partition.conf

With the above we can complete the transition of OEM tools and torcx to sysext.

Make sysext/syscfg work for more cases like kernel module settings or systemd configuration itself:

[ ] Run systemd-sysext for /sysroot in the initrd already, this can be tested in Flatcar (should run after initrd-setup-root-after-ignition.service) but this new unit should be upstreamed and as far as I know the upstream wish was also to support mounting /var through a kernel cmdline parameter to be able to account for sysext images under /var but not sure if this is a blocker: https://github.com/flatcar/Flatcar/issues/985
[ ] Support a mutable mode for systemd-sysext and systemd-syscfg. For systemd-sysext this mutable mode will help traditional distro users on Debian or Fedora become accustomed to systemd-sysext. For systemd-syscfg we could use this mutable mode for our /etc overlay mount, making /usr/share/flatcar/etc a syscfg directory and setting up a symlink to it from /usr/lib/syscfgs/. https://github.com/flatcar/Flatcar/issues/986
[ ] Make systemd-sysext replace the overlay mount atomically with Linux 6.5 "mount beneath", and then unmounting the old one with a regular lazy umount: https://github.com/flatcar/Flatcar/issues/1328
[ ] Add extension-release file setting to restart a list of given services (needed even with the daemon reload): https://github.com/flatcar/Flatcar/issues/1329
[x] As ergonomic improvement, systemd should support UpheldBy in the [Install] section and a TARGET.upholds dir with symlinks: https://github.com/flatcar/Flatcar/issues/987
[ ] Add systemd-sysext refresh --check-conflicts flag that prevents loading extensions when one extension would either shadow the base OS or another extension

This makes sysext/syscfg feature complete for advanced use and should also help to promote it outside of Flatcar.

Future ideas to use systemd-syscfg in Flatcar:
- Maybe leverage systemd-syscfg for OEM files that need to be under /etc - see https://github.com/flatcar/coreos-overlay/pull/2506#discussion_r1137984241 for a case where we would copy a file to /etc from a service and the file won't be updated → or better fix the lack of support for /usr for vendor-supplied configs if we encounter it!
- [ ] Use ignition-apply to bake a syscfg image, then one could replace an old syscfg with a newer one for live config updates without a reboot, given that, e.g., a small helper allows to embed the ignition-apply-baked syscfg in a final Ignition config to be used on first boot

flatcar / Flatcar

Roadmap: systemd-sysext and systemd-confext #978