search

coreos / ignition

First boot installer and configuration tool
https://coreos.github.io/ignition/
Apache License 2.0
828 stars 245 forks source link

Allow rebooting after Ignition if needed #1051

Open bgilbert opened 5 years ago

bgilbert commented 5 years ago

In particular, an Ignition config might want to configure kernel arguments for disabling SMT (https://github.com/coreos/fedora-coreos-tracker/issues/181). In that case, the first complete boot should have the same kargs as subsequent boots, so we'd want to reboot from the initramfs after Ignition runs.

(We can't have Ignition write a unit which reboots from inside the real root, since a half-completed first boot could cause a late-sequenced ConditionFirstBoot unit to never run at all.)

Ignition shouldn't have special handling for kernel arguments. Proposal: add an initramfs service which detects that certain relevant files have been written (e.g. for https://github.com/ostreedev/ostree/issues/479) and performs the reboot.

Filed here rather than in coreos/fedora-coreos-config because the mechanism can hopefully be generic. See also discussion in https://github.com/coreos/coreos-installer/issues/13.

bgilbert commented 5 years ago

Also, @cgwalters noted in https://github.com/coreos/coreos-installer/issues/13#issuecomment-493157500 that we'd need to mark Ignition complete from the initramfs rather than the real root.

dustymabe commented 5 years ago

👍 - seems reasonable to me. If we can now reboot I almost wonder if we couldn't take it 10 steps further and just let ignition replace the limited functionality of coreos-installer.

ajeddeloh commented 5 years ago

I'm in favor of this. If we do package layering from the initramfs (lets not talk about that here, its a wayyyys off) it would be useful there too.

If we go the "check if certain files changed" route for determining whether to reboot, we should include a flag file as well that's just for that which can be used as an escape hatch if needed or if someone has a different, unforeseen reason.

bgilbert commented 5 years ago

@dustymabe I think that'd involve something like https://github.com/coreos/fedora-coreos-tracker/issues/34. Seems like it could work like this:

  1. The user boots FCOS from PXE or a USB stick.
  2. Some unit converts the coreos.inst parameters to an Ignition config.
  3. Ignition runs, with the ostree being laid down between -disks and -files. The autogenerated Ignition config also fetches the real one and writes it into /boot.
  4. The system reboots and runs the real Ignition config.

We'd lose:

  1. The nice command-line tool that can be run from any distro.
  2. The ability to rerun the installer from the initramfs emergency shell if it fails.
  3. The ability to do other nice UX things like report progress.

I'm not certain that this approach is better than a separate installer, but it's worth considering at least.

bgilbert commented 5 years ago

@ajeddeloh The flag file might encourage people to toss in a reboot "for safety". I agree that we may end up needing one, but I'd be inclined to start without it and see what happens.

dustymabe commented 5 years ago

I'm not certain that this approach is better than a separate installer, but it's worth considering at least.

yeah I don't know if it's better, was just an idea I had.

ajeddeloh commented 5 years ago

@ajeddeloh The flag file might encourage people to toss in a reboot "for safety". I agree that we may end up needing one, but I'd be inclined to start without it and see what happens.

To play devil's advocate, they might start writing kargs as a way to trigger it. I'm fine leaving it off for the moment though, that's a good point.

cgwalters commented 5 years ago

OK I added a bunch of comments in https://github.com/coreos/coreos-installer/issues/13 but will summarize them here as it's a better place.

For RHCOS (OpenShift 4) the model is based on early pivot - we boot and then immediately upgrade and reboot. We also rely on changing kernel arguments at that time (to avoid double reboot).

Or to say it simply: We want to both upgrade and change kargs.

I am hesitating a lot about something ambitious like trying to do the upgrade from the initramfs. We will hit bootstrapping problems - for example, the RHCOS upgrade pulls from a container image, and we need the pull secret for that container. More generally, anything needed for upgrades needs to also be configured in the initramfs.

Plus dragging the whole rpm-ostree stack into the initramfs is a whole place where it's never been tested, and creates a whole special case for it that only runs one time.

The more I think about this, the more I feel like while it does make sense to drive Ignition/kargs integration, and perhaps more medium term spike on ostree-in-initramfs, short term we have other things we need to do first - such as getting RHCOS+MCO to Ignition Spec 3, and not to mention things like beefing up our CI story - before we can tackle fundamental reworks like this.

So my instinct now is to continue changing the MCD so that we handle kargs there, and keeping the way we do the early pivot rather than trying to do it in Ignition right now. But happy to continue to discuss!

bgilbert commented 5 years ago

The more I think about this, the more I feel like while it does make sense to drive Ignition/kargs integration, and perhaps more medium term spike on ostree-in-initramfs, short term we have other things we need to do first - such as getting RHCOS+MCO to Ignition Spec 3, and not to mention things like beefing up our CI story - before we can tackle fundamental reworks like this.

On the RHCOS side I think that makes sense. Re kargs I was thinking about the FCOS side, where we'll need this functionality and don't have an alternative in the MCO. If we can get https://github.com/ostreedev/ostree/issues/479 shipped, I think the initramfs part shouldn't be too major.

rfairley commented 5 years ago

Having a service makes sense to me. In https://github.com/ostreedev/ostree/issues/479 one thought is having something like /etc/ostree/kargs.d which contains snippets of karg key-value pairs. One way the reboot could be triggered is if /etc/ostree/kargs.d exists (an Ignition config should be provided that writes e.g. /etc/ostree/kargs.d/2000_localhost containing mykarg=foo anotherkarg=bar to configure custom kargs).

rfairley commented 5 years ago

One way the reboot could be triggered is if /etc/ostree/kargs.d exists

Also to point out - before the reboot, a redeployment (ostree admin deploy or equivalent) would need doing before the reboot as well, so that OSTree can write the new BLS config for the next boot based on the new /etc/ostree/kargs.d snippets (as the ostree kargs.d design currently stands).

rfairley commented 5 years ago

One way the reboot could be triggered is if /etc/ostree/kargs.d exists

Related thoughts: https://github.com/ostreedev/ostree/issues/479#issuecomment-505558298

rfairley commented 5 years ago

Related thoughts: ostreedev/ostree#479 (comment)

Follow up to this (I should have made that comment in this ticket): One approach for a way to determine whether Ignition added kargs could be to check the contents of /etc/ostree/kargs.d before and after ignition-files.service ran. If the contents changed, then assume Ignition added the kargs, and reboot the system.

zonggen commented 5 years ago

Current status