search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
260 stars 60 forks source link

Add factory reset capability #399

Open bgilbert opened 4 years ago

bgilbert commented 4 years ago

Under the immutable infrastructure model, we encourage users to handle config changes by reprovisioning their systems, rather than using configuration management or manually modifying the running system. This is okay in virtualized setups or in the cloud, and is also okay on bare metal where PXE infrastructure or IPMI-assisted ISO boot is available. But on single-node bare metal setups without external infrastructure, manual reinstall is impractical. (See e.g. this discussion.)

For this use case, it might help to have a command which receives an Ignition config to use for reprovisioning, copies it into /boot, sets first-boot kargs, and reboots. Then the initramfs, before running the usual first-boot steps, would need to restore the system to a pristine state. That process might look a lot like #94, or might look more like clearing /var and resetting /etc from /usr/etc. If those are in non-standard locations, perhaps the user-mode command could add kargs which tell the initramfs where to find them.

Because we'd be deleting user customizations before rerunning Ignition, this approach wouldn't[*] get into unsupported territory of using Ignition for configuration management. It would slightly complicate OS maintenance, though. At present we can freely change the Ignition initrd glue in ways that are incompatible with old installs (e.g. by changing first-boot kargs) because old installs will never run Ignition again. Supporting factory reset of old nodes would presumably add some additional constraints.

[*] Presumably it'd be infeasible to delete every possible user customization, so that statement might be optimistic.

bgilbert commented 4 years ago

Discussed in the community meeting today.

There was a general sense that the feature would be nice, but there was also concern about implementation complexity depending on the approach taken.

jlebon commented 4 years ago

Note with the stage 2 approach, as long as the stage 2 binary is the initrd we append to the base one, it should contain both the osmet file and the root squashfs, so it should be possible to do the re-install completely offline.

jlebon commented 4 years ago

Related OSTree issue: https://github.com/ostreedev/ostree/issues/1793

cgwalters commented 4 years ago

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

(This all said I think a lot of cases are going to be happy enough with the "ostree level" factory reset and not "wipe and replace disk image", particularly because the former can be made transactional)

cgwalters commented 3 years ago

As is today, it will work in many basic scenarios to just do:

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

A much stronger version would look like touch /run/factory-reset and we go back into the initramfs at shutdown time and do e.g. rm /var/* -rf && rsync -rlv --delete /usr/etc/ etc/.

The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved.

jlebon commented 3 years ago

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

Truly shipping the osmet file in the image is impossible because it's a circular dependency. So it would probably instead require coreos-installer to propagate it there manually at install time. But the problem is that the osmet file is tied to the OSTree commit of the aleph version, so if the node upgraded at all, we would need to fetch the aleph OSTree commit. One thing we could easily do is have coreos-installer add a ref to it it's not GC'ed, but then every node is paying that storage cost.

bgilbert commented 3 years ago

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot' $ reboot

This won't remove any existing customizations, so I don't think we should encourage it.

jlebon commented 2 years ago

A variation on https://github.com/coreos/fedora-coreos-tracker/issues/399#issuecomment-591713824 using kexec would be:

Since the rootfs CPIO includes the osmet file, no network is required during install. But air-gapped systems will need a way to obtain the rootfs initrd in the first place.

Would require some tweaks to support pointing at a local Ignition config from coreos.inst.* kargs, but if we're wrapping all this in a e.g. coreos-installer reinstall, then we could also just generate a live Ignition config and do away with coreos.inst.* kargs entirely.

jlebon commented 2 years ago

A variation on #399 (comment) using kexec would be:

Opened a proof of concept for this in https://github.com/coreos/coreos-installer/pull/712.

dustymabe commented 2 years ago

We discussed this in the community meeting today.

12:04:26    dustymabe | #agreed For our factory reset capability we'll use kexec and initially
                      | require either network access or local copies of the install media to
                      | exist. In the future we may generate intall media from the existing system.
                      | We also will limit our scope to just re-installing FCOS on FCOS and
                      | disallow/discourage running it from other distros.
lucab commented 2 years ago

Regarding kexec usage, man 7 kernel_lockdown says:

Only validly signed binaries may be kexec'd (waived if the binary image file to be executed is vouched for by IMA appraisal).

Kernel lockdown is automatically enabled when booting with UEFI SecureBoot, so I think that in practice we'll end up coupled with the lifecycle of the Fedora kernel signing key which is embedded in the currently booted FCOS kernel. There may be a way to load new keys for the target kernel signed by a different key, I didn't dig into that.

cgwalters commented 9 months ago

A new take on this in https://github.com/containers/bootc/pull/137

ericcurtin commented 4 months ago

I notice kexec was brought up, I only started learning this recently, while kexec works great on x86 platforms, there's some embedded platforms it doesn't work on. Apple Silicon/Asahi being one for example

ericcurtin commented 4 months ago

As is today, it will work in many basic scenarios to just do:

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

A much stronger version would look like touch /run/factory-reset and we go back into the initramfs at shutdown time and do e.g. rm /var/* -rf && rsync -rlv --delete /usr/etc/ etc/.

The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved.

I may look into this as it could be useful for an Automotive feature.

I think bootc will ultimately have the cleanest approach but it's hard to beat a neat tool like bootc that has an in-built installer right? :smile:

How about this as an option. As hinted in to the above approach as a more involved solution...

In ostree-prepare-root check some directory in the sysroot for a file (/run/factory-reset is fine that persists reboots right?)

If that file exists, in the C code during initrd do:

rm -rf var/*
rm -rf etc
cp -r usr/etc etc

and continue boot, more complex than this of course, but the above would be the gist of things.

This wouldn't be a live factory reset of course, but live resets are different.

jlebon commented 4 months ago

I notice kexec was brought up, I only started learning this recently, while kexec works great on x86 platforms, there's some embedded platforms it doesn't work on. Apple Silicon/Asahi being one for example

Possibly there's kinks that need to be worked out there specific to Apple Silicon, but AFAICT it is supported on aarch64 in general. We also have a kdump test which has no arch restriction (and so runs on all the arches we currently support).

cgwalters commented 4 months ago

It's already possible to do a form of this via ostree admin deploy --no-merge - and the really nice thing about that is it becomes trivial to carry forward state you do want from /etc by just copying it into the new deployment's /etc.

It's also transactional.

Resetting /var...hmm; I'm not sure it needs to be part of ostree, or at least not part of ostree-prepare-root.service which already does too much. The tricky thing here is offering some mechanism to only reset certain state.

I think I'd argue that this can be done in the real root during shutdown (after we've finalized the new bootloader entry). Something we likely should do in this scenario though is rerun through the new ostree logic to restore the factory /var state. If the /var cleanup happens in the real root then arbitrary complex tooling can operate on it using the full real root userspace.

There are some other bits of state though:

For these...it'd definitely be nice to retain the "aleph"/originally-installed state of these.