rawhide: ext.config.rpm-ostree.kernel-replace hangs in `ostree container encapsulate`

[dustymabe]

The most recent 40.20240103.91.0 run of rawhide fails in the ext.config.rpm-ostree.kernel-replace test because the ostree container encapsulate command just seems to hang forever:

[root@cosa-devsh ~]# bash foo.sh 
++ mktemp -d
+ cd /tmp/tmp.jlVF590aqc
+ image_dir=/var/tmp/fcos
+ image=oci:/var/tmp/fcos
+ image_pull=ostree-unverified-image:oci:/var/tmp/fcos
+ tmp_imagedir=/var/tmp/fcos-tmp
++ arch
+ arch=x86_64
+ kver=6.2.9-300.fc38.x86_64
+ rpm-ostree status --json
++ jq -r '.deployments[0].checksum'
+ checksum=ab312a2c757610b81ad2d2bff167ae8ea9508963185d093b218f45b381c6fdb5
++ jq -r '.deployments[0].version'
+ v0=40.20240103.dev.0
++ jq -r '.deployments[0]["container-image-reference"]'
+ imgref=ostree-remote-image:fedora:docker://quay.io/fedora/fedora-coreos:rawhide
+ rm /var/tmp/fcos -rf
+ encapsulate_args=()
+ [[ ostree-remote-image:fedora:docker://quay.io/fedora/fedora-coreos:rawhide != null ]]
+ encapsulate_args+=("--label" "ostree.bootable=true")
+ systemctl mask --now zincati
Created symlink /etc/systemd/system/zincati.service → /dev/null.
Warning: The unit file, source configuration file or drop-ins of zincati.service changed on disk. Run 'systemctl daemon-reload' to reload units.
+ ostree container encapsulate --label ostree.bootable=true --repo=/ostree/repo ab312a2c757610b81ad2d2bff167ae8ea9508963185d093b218f45b381c6fdb5 oci:/var/tmp/fcos

It is worth noting that for rawhide we deploy via container (https://github.com/coreos/fedora-coreos-config/pull/2711) so that may be part of the issue here.

[cgwalters]

BTW at a procedural level let's try to use labels (as I have now done) at least in addition to pinging specific people on chat for issues. The labels help ensure that someone else working ostree related things can find issues like this.

[dustymabe]

do you have notifications set up based on labels? I wasn't aware anything was really using them.