Open jlebon opened 3 years ago
we want "Ignition-like" semantics for the extensions: this means that the extensions ideally should be present from the get go in the real root on the first boot (or we should fail the boot if something goes wrong)
So one way to do this is to call out to rpm-ostree before switchroot to do the install and then doing livefs
(though I think we can do even better here since the sysroot isn't technically in use yet -- we'd need to enhance rpm-ostree for that).
A first cut would be to just reboot for now, the way we will do for kargs. This won't work for live ISO/PXE though.
For https://github.com/coreos/fedora-coreos-tracker/issues/767, note that these improvements should keep modular extensions in mind.
Just an idea: Afaik, *COS extensions today aren't something that RPM is aware of.
I wonder whether we could re-use Group
s defined in the spec for this (although I believe Group
s have been deprecated in RPM in Fedora)
Another path that came up in today's community meeting related to this is embedding packages at install time so that they can be layered offline on first boot. See https://github.com/coreos/fedora-coreos-tracker/issues/862#issuecomment-862575548.
Has there been any progress on this?
As an intermediate step to having Ignition support doing package layering in the initramfs we could take what is in the docs and modify it slightly and add it to the host. Then users could define their own RPMs in a dropin. Here's the full example:
variant: fcos
version: 1.3.0
storage:
files:
- path: /etc/systemd/system/rpm-ostree-install.service.d/rpms.conf
mode: 0644
contents:
inline: |
[Service]
Environment=RPMS="foo bar baz"
systemd:
units:
- name: rpm-ostree-install.service
enabled: true
contents: |
[Unit]
Description=Layer additional rpms
Wants=network-online.target
After=network-online.target
# We run before `zincati.service` to avoid conflicting rpm-ostree transactions.
Before=zincati.service
ConditionPathExists=!/var/lib/%N.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive $RPMS
ExecStart=/bin/touch /var/lib/%N.stamp
[Install]
WantedBy=multi-user.target
The proposal is that rpm-ostree-install.service
is shipped in the base layer of FCOS and the user only has to create the dropin with the RPMs they want to layer in their Butane/Ignition config.
Of course we can also add some sort of ConditionPathExists=
to make sure it only ran when users really wanted layered RPMs.
Feel free to ignore the message below. I realized this was quite inefficient since I was rebooting anyway so I've modified my approach to mirror the one above.
I was having some issues with rpm-ostree seemingly being killed/locked files (/etc/passwd) etc. There is already an issue about that somewhere on this issue tracker.
For now I've settled on using this for deployments:
variant: fcos
version: 1.5.0
storage:
directories:
- path: /var/cache/rpm-ostree-install
systemd:
units:
- name: rpm-ostree-install@.service
enabled: true
contents: |
[Unit]
Description=Layer %i with rpm-ostree
Wants=network-online.target
After=network-online.target
Before=zincati.service
ConditionPathExists=!/var/cache/rpm-ostree-install/%i.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --reboot --allow-inactive %i | tee /var/cache/rpm-ostree-install/%i.stamp"
[Install]
WantedBy=multi-user.target
- name: rpm-ostree-install@fish.service
enabled: true
- name: rpm-ostree-install@neovim.service
enabled: true
- name: rpm-ostree-install@htop.service
enabled: true
:wave: hi all, I've been experimenting with fcos and finding it really cool. the gap I'm trying to fill is core os install plus 2-3 packages which falls below the level of effort to making a respin distro. Having a service file works but seems unusual compared to other first boot systems like kickstart/anaconda or cloud-init which can install packages on first boot as part of their initialization routines.
Any update on this? It would be really great to have an easier way to auto-install packages on CoreOS without having to create systemd units, and to reduce the size and complexity of the butane YAML files
I've been trying to use the documented setup that uses a systemd unit on first boot, but in ~10 tries it only succeeded 2 times, the rest of the attempts failed with Could not resolve host: mirrors.fedoraproject.org
(even though the unit is set up to be run After=network-online.target
). It just feels like a hack, so I have given up on it for now.
It would be really awesome to have robust support for layering right in Ignition and thus Butane.
Hi. Yup, can confirm I’ve had that weird networking problem like that too. I don’t fully remember, but I think it was DNS related: you’ve got connectivity at the IP level, but some DNS caching in systemd-resolved is hitting you… or something. Not sure, it was a while ago.
Anyway, here is the line I added to the unit to make it work, just before the ExecStart=/usr/bin/rpm-ostree
line:
ExecStart=sh -c 'while ! ping -c 1 mirrors.fedoraproject.org; do sleep 1; done'
Hacky, sure :shrug:… Hope that helps someone !
Thanks, that makes the hack a bit more hacky but might actually work. :)
Another reason that makes these "systemd unit on first boot" hacks to hacks: If a system with layered packages is reprovisioned but the /var
filesystem is kept, the stamps will still be there, but the packages won't be. Ignition lacks a way of deleting files, so the stamps need to be deleted manually before the reprovisioning.
This is a follow-up to #401. Fedora CoreOS now ships with an archive repo which allows reliable package layering operations. However, there is still work remaining to make it easier to use. Right now, users have to create a systemd unit which runs
rpm-ostree install --reboot ...
which is not great.Specifically:
The sugar that was proposed in #401 was something like: