replace-systemd: example boots but systemctl can't find shared library.

[jmarrero]

systemctl: error while loading shared libraries: libsystemd-shared-251.7-611.fc37.so: cannot open shared object file: No such file or directory

I updated the example because it was no longer building but to build I needed to remove /var/libs

/var/libs content I imagine was important?

FROM quay.io/fedora/fedora-coreos:stable
RUN rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2022-bb55f82158 && \
    #https://coreos.github.io/rpm-ostree/architecture-core/#content-in-var
    echo "ls /var/lib\n" && ls /var/lib/ && \
    ostree container commit

STEP 1/2: FROM quay.io/fedora/fedora-coreos:stable
STEP 2/2: RUN rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2022-bb55f82158 &&     echo "ls /var/lib\n" && ls /var/lib/ &&     ostree container commit
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-udev-251.6-609.fc37.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-251.6-609.fc37.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-libs-251.6-609.fc37.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-container-251.6-609.fc37.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-pam-251.6-609.fc37.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org/packages/systemd/251.6/609.fc37/x86_64/systemd-resolved-251.6-609.fc37.x86_64.rpm...done
Enabled rpm-md repositories: fedora updates-modular updates fedora-cisco-openh264 fedora-modular updates-archive
Updating metadata for 'fedora'...done
Updating metadata for 'updates-modular'...done
Updating metadata for 'updates'...done
Updating metadata for 'fedora-cisco-openh264'...done
Updating metadata for 'fedora-modular'...done
Updating metadata for 'updates-archive'...done
Importing rpm-md...done
rpm-md repo 'fedora'; generated: 2022-11-10T09:30:00Z solvables: 66822
rpm-md repo 'updates-modular'; generated: 2022-12-09T00:53:59Z solvables: 1457
rpm-md repo 'updates'; generated: 2022-12-15T01:21:14Z solvables: 12985
rpm-md repo 'fedora-cisco-openh264'; generated: 2022-10-06T11:01:40Z solvables: 4
rpm-md repo 'fedora-modular'; generated: 2022-11-10T09:23:24Z solvables: 1454
rpm-md repo 'updates-archive'; generated: 2022-12-15T02:02:37Z solvables: 12315
Resolving dependencies...done
Installing 6 packages:
  systemd-251.6-609.fc37.x86_64 (@commandline)
  systemd-container-251.6-609.fc37.x86_64 (@commandline)
  systemd-libs-251.6-609.fc37.x86_64 (@commandline)
  systemd-pam-251.6-609.fc37.x86_64 (@commandline)
  systemd-resolved-251.6-609.fc37.x86_64 (@commandline)
  systemd-udev-251.6-609.fc37.x86_64 (@commandline)
Downgrading: systemd-libs;251.6-609.fc37;x86_64;local
Downgrading: systemd-pam;251.6-609.fc37;x86_64;local
Downgrading: systemd-resolved;251.6-609.fc37;x86_64;local
Downgrading: systemd;251.6-609.fc37;x86_64;local
Downgrading: systemd-container;251.6-609.fc37;x86_64;local
Downgrading: systemd-udev;251.6-609.fc37;x86_64;local
Cleanup: systemd-udev;251.7-611.fc37;x86_64;installed
Cleanup: systemd-container;251.7-611.fc37;x86_64;installed
Cleanup: systemd;251.7-611.fc37;x86_64;installed
Cleanup: systemd-libs;251.7-611.fc37;x86_64;installed
Cleanup: systemd-pam;251.7-611.fc37;x86_64;installed
Cleanup: systemd-resolved;251.7-611.fc37;x86_64;installed
/usr/lib/sysusers.d/basic.conf:9: conflict with earlier configuration for user 'root', ignoring line.
/usr/lib/sysusers.d/basic.conf:12: conflict with earlier configuration for group 'nobody', ignoring line.
/usr/lib/sysusers.d/basic.conf:13: conflict with earlier configuration for user 'nobody', ignoring line.
/usr/lib/sysusers.d/chrony.conf:2: conflict with earlier configuration for user 'chrony', ignoring line.
/usr/lib/sysusers.d/systemd-resolve.conf:8: conflict with earlier configuration for user 'systemd-resolve', ignoring line.
/usr/lib/sysusers.d/systemd-timesync.conf:8: conflict with earlier configuration for user 'systemd-timesync', ignoring line.
ls /var/lib\n
NetworkManager
alternatives
authselect
chrony
containerd
containers
dnsmasq
fwupd
games
initramfs
iscsi
kdump
logrotate
machines
misc
nfs
os-prober
polkit-1
portables
private
rpm
samba
selinux
sss
systemd
tpm2-tss
vagrant
Found file: "var/lib/systemd/catalog/database"
Found file: "var/lib/systemd/random-seed"
Found file: "var/lib/alternatives"
Found file: "var/lib/rpm"
Found file: "var/lib/vagrant"
Found file: "var/lock"
Found file: "var/log/README"
Found file: "var/log/tallylog"
Found file: "var/log/wtmp"
Found file: "var/log/btmp"
Found file: "var/log/lastlog"
Found file: "var/mail"
Found file: "var/run"
error: Found content in var

[jmarrero]

Any quick and fast way to resolve this or remove the example for now?

[jmarrero]

No luck with an upgrade using: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a82842a059 either:

Last login: Thu Dec 15 20:40:26 on ttyS0
Fedora CoreOS 37.20221106.3.0
systemctl: error while loading shared libraries: libsystemd-shared-251.7-611.fc37.so: cannot open shared object file: No such file or directory
[core@tutorial ~]$ sudo reboot
reboot: error while loading shared libraries: libsystemd-shared-251.7-611.fc37.so: cannot open shared object file: No such file or directory
[core@tutorial ~]$ 

[cgwalters]

What seems to be happening is that for some reason the /usr/bin/systemctl is left at the old version. Doing rpm --reinstall https://kojipkgs.fedoraproject.org//packages/systemd/251.6/609.fc37/x86_64/systemd-251.6-609.fc37.x86_64.rpm fixes it here.

# rpm -V systemd|grep -Fv '.......T.' | grep -v missing
....L....    /usr/lib/systemd/system/default.target
# rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2022-bb55f82158
...
(apparent success)
...
# sha256sum /usr/bin/systemctl
acac0aa21c430381bf7e4bcb48f1e85d8d1cd0530b6ee682e1d47cee725aefd4  /usr/bin/systemctl
# rpm -V systemd|grep -Fv '.......T.' | grep -v missing
..5....T.    /usr/bin/systemctl
# 

So it's just somehow only that binary...

[cgwalters]

Oh man, of course...the problem is our cliwrap logic again...we're swapping it back to the old binary. To handle this, I think we may need to detect the case where /usr/bin/systemctl got swapped back, and then just delete the old version.

[jmarrero]

thanks for the workaround here is in a PR: https://github.com/coreos/layering-examples/pull/50 Lets leave this issue open while we get a proper fix in rpm-ostree to remove the workaround then.