Document automated reprovisioning for bare metal

[bgilbert]

We want to encourage users to reprovision their bare-metal nodes when their config changes, but we don't document how to do it. In the long run, https://github.com/coreos/fedora-coreos-tracker/issues/399 might be the right approach, but in the short term, other mechanisms are required. This is arguably part of https://github.com/coreos/fedora-coreos-docs/issues/117, but I'm filing it separately to distinguish the higher-level workflow issue from the nuts-and-bolts topics in #117.

There are a few possible reprovisioning flows:

• UEFI + PXE: use efibootmgr --bootnext to switch the machine into netboot mode just long enough to reinstall the node. (I haven't actually tried this.)
• PXE generally: set the machine to always netboot, but have the netboot server redirect the boot back to the local disk unless it's been told to reprovision this particular machine. Does Matchbox help here?
• IPMI + virtual CD + ISO: use coreos-installer iso kargs embed to set coreos.inst.install_dev= and related kargs, then push the ISO to the system with IPMI.
• Just run from live ISO / PXE and never install to disk. This requires some other mechanism for OS updates.

Whichever flows we decide to recommend, we should provide step-by-step instructions.

[bgilbert]

We should also document how to preserve data volumes when reprovisioning. The traditional model is to do this with Ignition, but there are some quirks and they're not documented in one place:

• storage.disks: specify the data partition(s); set wipe_table: false and wipe_partition_entry: false. On secondary disks, Ignition will leave existing partitions alone. On the boot disk, coreos-installer will wipe the partition table entries (but not the data) and then Ignition will recreate them with the same parameters as before.
• storage.raid: probably safest to avoid, as Ignition always recreates the RAID. Recreating a RAID with the same parameters might preserve data, but I don't know if this has been tested.
• storage.luks: specify the data volumes; set wipe_volume: false and use a static key_file. Ignition will reuse the existing LUKS volumes, or create new ones if the backing storage is blank. Note that Ignition can't reuse LUKS volumes encrypted with ephemeral keys or Clevis. The static key will be stored on the root filesystem, which can be encrypted with Clevis if desired.
• storage.filesystems: specify the filesystems and set wipe_filesystem: false. Ignition will reuse existing filesystems, or create new ones if the backing storage is blank.

This procedure assumes the partition layout in the new Ignition config matches the old one. If not, Ignition may clobber the data volumes. It also assumes that the original layout left at least the documented 8 GiB for the root filesystem; if not, reprovisioning with a newer OS version may clobber the first data volume on the boot disk due to coreos/fedora-coreos-tracker#586.

For additional safety, coreos-installer provides a partition-saving mechanism (--save-part{label,index} / coreos.inst.save_part{label,index}) which explicitly preserves specified partitions during a reinstall. This is safe against storage.disks desynchronization (since Ignition won't clobber the saved partitions unless told to) and against coreos/fedora-coreos-tracker#586 (since coreos-installer won't overwrite a saved partition). Partition saving uses the same Ignition configs as in the traditional model; for fresh installs, coreos-installer will ignore requests to save partitions that don't exist. Downsides of this approach are that the installer needs to be explicitly told about data partitions that are expected to exist on the boot disk, and that fresh installs may need to explicitly wipe the boot disk to avoid preserving old data.

[stereobutter]

We are currently working on a POC of using FCOS for IIoT in industrial automation. The device provisioning is currently done via iso images with embedded ignition config and we are quite happy with this, as this is much easier to setup than a working PXE environment.

This is doubly true for reprovisioning devices in the field where we don't control the network at all and rather (a) interactively reprovision a machine by executing some flavor of coreos-installer install ... over ssh or (b) prepare a magic usb stick that does the same.

From what I gather this is already possible using coreos-installer as is (using the partition saving mechanism etc.)? In any case an explicit api (maybe as part of coreos-installer) that resets/reprovisions a machine (with some options to control what to keep and what to reset/override/append) would be a welcome addition.

[bgilbert]

@stereobutter There's some POC work happening in https://github.com/coreos/coreos-installer/pull/712. For the "magic USB stick" case, you may also be interested in the new coreos-installer iso customize command, which should make such images easier to create.

[runiq]

storage.raid: probably safest to avoid, as Ignition always recreates the RAID. Recreating a RAID with the same parameters might preserve data, but I don't know if this has been tested.

It doesn't preserve data if a disk has been replaced, I tested.