Improve package layering UX

[jlebon]

This is a follow-up to #401. Fedora CoreOS now ships with an archive repo which allows reliable package layering operations. However, there is still work remaining to make it easier to use. Right now, users have to create a systemd unit which runs rpm-ostree install --reboot ... which is not great.

Specifically:

1. we need a better interface for package layering via Ignition, whether that be via file drop-ins translated by FCOS glue code, or drop-ins read by rpm-ostree directly (related: https://github.com/coreos/rpm-ostree/issues/2326)
2. we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)
3. we need an FCC-friendly way to specify these packages, either directly having users use the file drop-in interface or via sugar

The sugar that was proposed in #401 was something like:

extensions:
- usbguard

[jlebon]

we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)

So one way to do this is to call out to rpm-ostree before switchroot to do the install and then doing livefs (though I think we can do even better here since the sysroot isn't technically in use yet -- we'd need to enhance rpm-ostree for that).

A first cut would be to just reboot for now, the way we will do for kargs. This won't work for live ISO/PXE though.

[jlebon]

For https://github.com/coreos/fedora-coreos-tracker/issues/767, note that these improvements should keep modular extensions in mind.

[LorbusChris]

Just an idea: Afaik, *COS extensions today aren't something that RPM is aware of. I wonder whether we could re-use Groups defined in the spec for this (although I believe Groups have been deprecated in RPM in Fedora)

[jlebon]

Another path that came up in today's community meeting related to this is embedding packages at install time so that they can be layered offline on first boot. See https://github.com/coreos/fedora-coreos-tracker/issues/862#issuecomment-862575548.

[alvarlagerlof]

Has there been any progress on this?

[dustymabe]

As an intermediate step to having Ignition support doing package layering in the initramfs we could take what is in the docs and modify it slightly and add it to the host. Then users could define their own RPMs in a dropin. Here's the full example:

variant: fcos
version: 1.3.0
storage:
  files:
    - path: /etc/systemd/system/rpm-ostree-install.service.d/rpms.conf
      mode: 0644
      contents:
        inline: |
          [Service]
          Environment=RPMS="foo bar baz"
systemd:
  units:
    - name: rpm-ostree-install.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer additional rpms
        Wants=network-online.target
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive $RPMS
        ExecStart=/bin/touch /var/lib/%N.stamp
        [Install]
        WantedBy=multi-user.target

The proposal is that rpm-ostree-install.service is shipped in the base layer of FCOS and the user only has to create the dropin with the RPMs they want to layer in their Butane/Ignition config.

Of course we can also add some sort of ConditionPathExists= to make sure it only ran when users really wanted layered RPMs.

[xanderificnl]

Feel free to ignore the message below. I realized this was quite inefficient since I was rebooting anyway so I've modified my approach to mirror the one above.

---

I was having some issues with rpm-ostree seemingly being killed/locked files (/etc/passwd) etc. There is already an issue about that somewhere on this issue tracker.

For now I've settled on using this for deployments:

• prevents double running rpm-ostree-installer@'s
• reboots after every install (considering it's only intended for new deployments, I felt it was acceptable) to prevent the /etc/passwd lockings

variant: fcos
version: 1.5.0
storage:
  directories:
    - path: /var/cache/rpm-ostree-install
systemd:
  units:
   - name: rpm-ostree-install@.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer %i with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/cache/rpm-ostree-install/%i.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --reboot --allow-inactive %i | tee /var/cache/rpm-ostree-install/%i.stamp"

        [Install]
        WantedBy=multi-user.target
    - name: rpm-ostree-install@fish.service
      enabled: true
    - name: rpm-ostree-install@neovim.service
      enabled: true
    - name: rpm-ostree-install@htop.service
      enabled: true

[Mossman1215]

:wave: hi all, I've been experimenting with fcos and finding it really cool. the gap I'm trying to fill is core os install plus 2-3 packages which falls below the level of effort to making a respin distro. Having a service file works but seems unusual compared to other first boot systems like kickstart/anaconda or cloud-init which can install packages on first boot as part of their initialization routines.

[usp-npe]

Any update on this? It would be really great to have an easier way to auto-install packages on CoreOS without having to create systemd units, and to reduce the size and complexity of the butane YAML files

[lukasbestle]

I've been trying to use the documented setup that uses a systemd unit on first boot, but in ~10 tries it only succeeded 2 times, the rest of the attempts failed with Could not resolve host: mirrors.fedoraproject.org (even though the unit is set up to be run After=network-online.target). It just feels like a hack, so I have given up on it for now.

It would be really awesome to have robust support for layering right in Ignition and thus Butane.

[navaati]

Hi. Yup, can confirm I’ve had that weird networking problem like that too. I don’t fully remember, but I think it was DNS related: you’ve got connectivity at the IP level, but some DNS caching in systemd-resolved is hitting you… or something. Not sure, it was a while ago.

Anyway, here is the line I added to the unit to make it work, just before the ExecStart=/usr/bin/rpm-ostree line:

      ExecStart=sh -c 'while ! ping -c 1 mirrors.fedoraproject.org; do sleep 1; done'

Hacky, sure :shrug:… Hope that helps someone !

[lukasbestle]

Thanks, that makes the hack a bit more hacky but might actually work. :)

[lukasbestle]

Another reason that makes these "systemd unit on first boot" hacks to hacks: If a system with layered packages is reprovisioned but the /var filesystem is kept, the stamps will still be there, but the packages won't be. Ignition lacks a way of deleting files, so the stamps need to be deleted manually before the reprovisioning.