support in-place updates of extensions

[miabbott]

On Fedora 33 SB, using rpm-ostree-2021.2-2 as root:

I had built an ostree commit and then an extensions ostree commit. I changed extensions.yaml to include a new extension and rpm-ostree barked at me:

# rpm-ostree compose tree --unified-core --repo=repo --cachedir=cache os/manifest.yaml
...
Wrote commit: f76ffb5c41592e7b56aee87c3e9f2c6dfa371a325a97b937c80f8d0d6efa696b
Metadata Total: 10479
Metadata Written: 26
Content Total: 6250
Content Written: 27
Content Cache Hits: 21135
Content Bytes Written: 128935778

# rpm-ostree compose extensions --repo=repo --cachedir=cache os/manifest.yaml os/extensions.yaml --output-dir=extensions --base-rev=f76ffb5c41592e7b56aee87c3e9f2c6dfa371a325a97b937c80f8d0d6efa696b
Checking out f76ffb5... done!
warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
Enabled rpm-md repositories: rhel-8-server-ose rhel-8-fast-datapath rhel-8-baseos rhel-8-appstream rhel-8-nfv rhel-8-advanced-virt
rpm-md repo 'rhel-8-server-ose' (cached); generated: 2021-03-04T17:12:14Z
rpm-md repo 'rhel-8-fast-datapath' (cached); generated: 2021-02-15T16:31:03Z
rpm-md repo 'rhel-8-baseos' (cached); generated: 2021-03-04T07:24:48Z
rpm-md repo 'rhel-8-appstream' (cached); generated: 2021-03-04T15:59:58Z
rpm-md repo 'rhel-8-nfv' (cached); generated: 2021-02-16T13:51:51Z
rpm-md repo 'rhel-8-advanced-virt' (cached); generated: 2021-02-25T13:14:46Z
⠁ Importing rpm-md   0% [░░░░░░░░░░░░░░░░░░░░] (0s)
⠲ Importing rpm-md  83% [████████████████░░░░] (0s)

Importing rpm-md... done
Resolving dependencies... done
error: linkat: File exists

# ls -latr extensions/
total 174964
-rw-r--r--. 1 root root  5257628 Mar  4 13:32 kernel-rt-modules-extra-4.18.0-240.15.1.rt7.69.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 17469888 Mar  4 13:32 kernel-rt-devel-4.18.0-240.15.1.rt7.69.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 31210224 Mar  4 13:32 kernel-rt-core-4.18.0-240.15.1.rt7.69.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 26833148 Mar  4 13:32 kernel-rt-modules-4.18.0-240.15.1.rt7.69.el8_3.x86_64.rpm
-rw-r--r--. 1 root root  4993288 Mar  4 13:32 kernel-rt-kvm-4.18.0-240.15.1.rt7.69.el8_3.x86_64.rpm
-rw-r--r--. 1 root root    27724 Mar  4 13:32 usbguard-selinux-0.7.8-7.el8.noarch.rpm
-rw-r--r--. 1 root root   539756 Mar  4 13:32 usbguard-0.7.8-7.el8.x86_64.rpm
-rw-r--r--. 1 root root   913424 Mar  4 13:32 protobuf-3.5.0-13.el8.x86_64.rpm
-rw-r--r--. 1 root root   115684 Mar  4 13:32 libqb-1.0.3-12.el8.x86_64.rpm
-rw-r--r--. 1 root root  5830492 Mar  4 13:32 kernel-headers-4.18.0-240.15.1.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 17449820 Mar  4 13:32 kernel-devel-4.18.0-240.15.1.el8_3.x86_64.rpm
-rw-r--r--. 1 root root  5233868 Mar  4 13:32 kernel-modules-extra-4.18.0-240.15.1.el8_3.x86_64.rpm
-rw-r--r--. 1 root root  4555608 Mar  4 13:32 kernel-4.18.0-240.15.1.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 26813336 Mar  4 13:32 kernel-modules-4.18.0-240.15.1.el8_3.x86_64.rpm
-rw-r--r--. 1 root root 31867892 Mar  4 13:32 kernel-core-4.18.0-240.15.1.el8_3.x86_64.rpm
drwxr-xr-x. 6 root root       59 Mar  4 13:32 ..
-rw-r--r--. 1 root root      128 Mar  4 13:33 .rpm-ostree-state-chksum
-rw-r--r--. 1 root root      951 Mar  4 13:33 extensions.json
drwxr-xr-x. 2 root root     4096 Mar  4 13:33 .

[jlebon]

This... did work at some point during development. :)

I started looking at this, but first I need to make it smarter so it doesn't even try to download packages which are already present. At least cosa won't hit this because it always nukes the output dir first.

[LukeShortCloud]

Hey @jlebon , I recently ran into this (or, at least, a similar issue since I am not using extensions) when trying to compose an rpm-ostree tree with the wine package. Creating a hard link is not working. I get this error: linkat: File exists

The issue I ran into is actually with the unixODBC.i686 package but I tried (1) only installing unixODBC.i686, (2) only installing unixODBC.x86_64, and (3) installing both of them. The issue never showed up doing any of those. It only showed up when installing wine.

Commands to reproduce:

git clone --branch f38 https://pagure.io/workstation-ostree-config.git
cd workstation-ostree-config
export WORKDIR="/root/tmp"
sudo -E mkdir -p ${WORKDIR}/cache ${WORKDIR}/repo
sudo -E ostree --repo=${WORKDIR}/repo init --mode=archive-z2
cat <<EOF > fedora-silverblue-unixodbc.yaml
---
include: fedora-silverblue.yaml
releasever: "38"
packages:
  - wine
EOF
sudo -E rpm-ostree compose tree --unified-core --cachedir=${WORKDIR}/cache --repo=${WORKDIR}/repo fedora-silverblue-unixodbc.yaml

Last few lines of output:

Input state hash: 6b84d1d1bad57c8bbf3903b1760e1fa8740126e7eac6163272b56b71e4295a74
Relabeling...done
Checking out packages...done
error: Checkout unixODBC-2.3.11-2.fc38.i686: Copy checkout of 4d252bc079de54d3fbb732ca336c66eca45c6ae6704ed5761480ff87a5313466 to odbc.ini: linkat: File exists

As a short-term workaround, I can do a build without --unified-core but then it tells me that not using it is deprecated.

NOTICE: Running rpm-ostree compose tree without --unified-core is deprecated.
 Please add --unified-core to the command line and ensure your content
 works with it.  For more information, see https://github.com/coreos/rpm-ostree/issues/729

Potentially related:

• https://github.com/coreos/rpm-ostree/issues/1582
• https://github.com/coreos/rpm-ostree/issues/1937

[LukeShortCloud]

Actually, for my production build, I cannot get rid of --unified-core because I am using modules.

NOTICE: treefile key `modules` is experimental and subject to change

rpm-ostree version: 2023.7
error: Composing with modules requires --unified-core

It sounds like modularity in Fedora is going to be removed soon. I am not sure if that is just for Fedora itself or for all managed DNF repositories long-term, too. As another workaround, I can download the packages and re-host them as a non-modular repository.

https://fedoraproject.org/wiki/Changes/RetireModularity

[jlebon]

@LukeShortCloud That seems like a separate bug. Can you file a new issue?