Aurora-dx-nvidia No Longer Automatically Applying Updates After Reboot

Mortifier13 commented 2 months ago

Describe the bug

I have 2 systems (desktop and laptop) running aurora-dx-nvidia on different branches - latest and stable. I’ve noticed in the last few weeks ((around the 7th or so), the updates are downloading but they are not applying after reboot. I have to manually rollback to a previous version (the 0801 one) then do a rpm-ostree update in order for updates to apply on reboot. I had to do that with the laptop on stable last week and it just downloaded a new update for this week and will not apply it automatically. Here is the rpm-ostree status after rebooting when it did not apply:

What did you expect to happen?

❯ rpm-ostree status -v State: idle AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot Deployments: ● ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable (index: 0) Digest: sha256:3c79afa3f5dbb3f8db4e0a4b243b80db31b5b689b63e781094e31dc20000fb5e Version: 40.20240813.0 (2024-08-13T06:35:52Z) Commit: 48c1d27ab8d7b8cb967f1e4aa906abf1b2eb65adf672ee5825f69bfd6a251154 Staged: no StateRoot: fedora

ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable (index: 1) Digest: sha256:392c885695d7b8e4c89c42ee21e94c0f8f86647ffdb343dd2c99c21ae397432f Version: 40.20240801.0 (2024-08-01T19:56:00Z) Commit: 2ad3671f2be2b081e4048533e18c120682684f5b6a7e7145b64fe80bc0051dea StateRoot: fedora

AvailableUpdate: Timestamp: 2024-08-20T05:52:32Z Commit: 6d82f58c00667266c94380faa7ff7f75ca2b134330dcb8d7246478a6f4d8f782 Upgraded: amd-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40 amd-ucode-firmware 20240709-1.fc40 → 20240811-2.fc40 atheros-firmware 20240709-1.fc40 → 20240811-2.fc40 brcmfmac-firmware 20240709-1.fc40 → 20240811-2.fc40 breeze-icon-theme 6.5.0-1.fc40 → 6.5.0-2.fc40 btrfs-progs 6.9.2-1.fc40 → 6.10-1.fc40 buildah 1.36.0-1.fc40 → 1.37.0-1.fc40 cirrus-audio-firmware 20240709-1.fc40 → 20240811-2.fc40 cockpit-machines 316-1.fc40 → 317-1.fc40 cockpit-ostree 1:203-1.fc40 → 1:204-1.fc40 cockpit-podman 91-1.fc40 → 92-1.fc40 code 1.92.1-1723066391.el8 → 1.92.2-1723661076.el8 devpod v0.5.18-1.fc40 → v0.5.19-1.fc40 distribution-gpg-keys 1.104-1.fc40 → 1.105-1.fc40 docker-buildx-plugin 0.16.1-1.fc40 → 0.16.2-1.fc40 docker-ce 3:27.1.1-1.fc40 → 3:27.1.2-1.fc40 docker-ce-cli 1:27.1.1-1.fc40 → 1:27.1.2-1.fc40 docker-ce-rootless-extras 27.1.1-1.fc40 → 27.1.2-1.fc40 egl-gbm 2:1.1.1-5.20240412git649c079.fc40 → 2:1.1.2-1.fc40 egl-gbm 2:1.1.1-5.20240412git649c079.fc40 → 2:1.1.2-1.fc40 egl-wayland 1.1.15-1.fc40 → 1.1.15-2.20240814gitf30cb0e.fc40 egl-wayland 1.1.15-1.fc40 → 1.1.15-2.20240814gitf30cb0e.fc40 ethtool 2:6.9-1.fc40 → 2:6.10-1.fc40 fzf 0.54.1-1.fc40 → 0.54.3-1.fc40 gperftools-libs 2.14-3.fc40 → 2.14-4.fc40 ibus 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-gtk2 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-gtk3 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-gtk4 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-libs 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-panel 1.5.30-5.fc40 → 1.5.30-6.fc40 ibus-setup 1.5.30-5.fc40 → 1.5.30-6.fc40 intel-audio-firmware 20240709-1.fc40 → 20240811-2.fc40 intel-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40 intel-media-driver 24.1.5-1.fc40 → 24.2.5-1.fc40 iwlegacy-firmware 20240709-1.fc40 → 20240811-2.fc40 iwlwifi-dvm-firmware 20240709-1.fc40 → 20240811-2.fc40 iwlwifi-mvm-firmware 20240709-1.fc40 → 20240811-2.fc40 kcli 99.0.0.git.202408112231.ec8c845-0.fc40 → 99.0.0.git.202408152041.5eeaa03-0.fc40 kde-settings 40.0-1.fc40 → 40.1-1.fc40 kde-settings-plasma 40.0-1.fc40 → 40.1-1.fc40 kde-settings-pulseaudio 40.0-1.fc40 → 40.1-1.fc40 kde-settings-sddm 40.0-1.fc40 → 40.1-1.fc40 kernel 6.9.7-200.fc40 → 6.9.11-200.fc40 kernel-core 6.9.7-200.fc40 → 6.9.11-200.fc40 kernel-modules 6.9.7-200.fc40 → 6.9.11-200.fc40 kernel-modules-core 6.9.7-200.fc40 → 6.9.11-200.fc40 kernel-modules-extra 6.9.7-200.fc40 → 6.9.11-200.fc40 kernel-tools 6.10.3-200.fc40 → 6.10.5-200.fc40 kernel-tools-libs 6.10.3-200.fc40 → 6.10.5-200.fc40 kf6-breeze-icons 6.5.0-1.fc40 → 6.5.0-2.fc40 libedit 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40 libedit 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40 libedit-devel 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40 libertas-firmware 20240709-1.fc40 → 20240811-2.fc40 libimagequant 4.0.3-3.fc40 → 4.0.3-5.fc40 libldb 2.9.1-1.fc40 → 2.9.1-4.fc40 libnfsidmap 1:2.6.4-0.rc6.fc40 → 1:2.6.4-0.rc8.fc40 libppd 1:2.0.0-4.fc40 → 1:2.0.0-6.fc40 libsrtp 2.3.0-14.fc40 → 2.6.0-1.fc40 linux-firmware 20240709-1.fc40 → 20240811-2.fc40 linux-firmware-whence 20240709-1.fc40 → 20240811-2.fc40 mt7xxx-firmware 20240709-1.fc40 → 20240811-2.fc40 nfs-utils 1:2.6.4-0.rc6.fc40 → 1:2.6.4-0.rc8.fc40 nvidia-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40 nxpwireless-firmware 20240709-1.fc40 → 20240811-2.fc40 openssl 1:3.2.1-2.fc40 → 1:3.2.2-3.fc40 openssl-libs 1:3.2.1-2.fc40 → 1:3.2.2-3.fc40 passt 0^20240726.g57a21d2-1.fc40 → 0^20240814.g61c0b0d-1.fc40 passt-selinux 0^20240726.g57a21d2-1.fc40 → 0^20240814.g61c0b0d-1.fc40 podman 5:5.1.2-1.fc40 → 5:5.2.0-1.fc40 podmansh 5:5.1.2-1.fc40 → 5:5.2.0-1.fc40 python3-boto3 1.34.153-1.fc40 → 1.34.162-1.fc40 python3-botocore 1.34.153-1.fc40 → 1.34.162-1.fc40 qt-settings 40.0-1.fc40 → 40.1-1.fc40 realtek-firmware 20240709-1.fc40 → 20240811-2.fc40 rpm-ostree 2024.6-1.fc40 → 2024.7-1.fc40 rpm-ostree-libs 2024.6-1.fc40 → 2024.7-1.fc40 skopeo 1:1.15.2-1.fc40 → 1:1.16.0-1.fc40 tailscale 1.70.0-1.fc40 → 1.72.0-1 tiwilink-firmware 20240709-1.fc40 → 20240811-2.fc40 vim-common 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 vim-data 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 vim-enhanced 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 vim-filesystem 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 vim-minimal 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 wpa_supplicant 1:2.11-2.fc40 → 1:2.11-3.fc40 xxd 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40 Removed: displaylink-6.0.0-2.fc40.x86_64 gvisor-tap-vsock-6:0.7.4-1.fc40.x86_64 gvisor-tap-vsock-gvforwarder-6:0.7.4-1.fc40.x86_64 kmod-evdi-6.9.7-200.fc40.x86_64-1.14.5-2.20240726giteab561a.fc40.x86_64 kmod-kvmfr-6.9.7-200.fc40.x86_64-0.0.git.23.2de42028-1.fc40.x86_64 kmod-nvidia-6.9.7-200.fc40.x86_64-3:560.31.02-1.fc40.x86_64 kmod-openrazer-6.9.7-200.fc40.x86_64-100.0.0.git.530.886f986d-1.fc40.x86_64 kmod-v4l2loopback-6.9.7-200.fc40.x86_64-0.13.1-1.fc40.x86_64 kmod-wl-6.9.7-200.fc40.x86_64-6.30.223.271-51.fc40.x86_64 kmod-xone-6.9.7-200.fc40.x86_64-0.0.git.115.fdbb71f1-1.fc40.x86_64 kmod-xpadneo-6.9.7-200.fc40.x86_64-0.9.6-2.20240423git73be2eb.fc40.x86_64 kmod-zfs-6.9.7-200.fc40.x86_64-2.2.5-1.fc40.x86_64 libevdi-1.14.5-2.20230726giteab561a.fc40.x86_64 Added: kmod-kvmfr-6.9.11-200.fc40.x86_64-0.0.git.23.2de42028-1.fc40.x86_64 kmod-nvidia-6.9.11-200.fc40.x86_64-3:560.31.02-1.fc40.x86_64 kmod-openrazer-6.9.11-200.fc40.x86_64-100.0.0.git.530.886f986d-1.fc40.x86_64 kmod-v4l2loopback-6.9.11-200.fc40.x86_64-0.13.1-1.fc40.x86_64 kmod-wl-6.9.11-200.fc40.x86_64-6.30.223.271-51.fc40.x86_64 kmod-xone-6.9.11-200.fc40.x86_64-0.0.git.115.fdbb71f1-1.fc40.x86_64 kmod-xpadneo-6.9.11-200.fc40.x86_64-0.9.6-2.20240423git73be2eb.fc40.x86_64 kmod-zfs-6.9.11-200.fc40.x86_64-2.2.5-1.fc40.x86_64

~ ❯

Output of `rpm-ostree status`

No response

Output of `groups`

No response

Extra information or context

No response

lion7 commented 2 months ago

My automatic updates on Aurora stopped working as well, with the following error:

❯ rpm-ostree upgrade 
note: automatic updates (stage) are enabled
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: cryptographic signature verification failed: invalid signature when validating ASN.1 encoded signature

I think this is due to https://github.com/ublue-os/main/issues/599 and https://github.com/ublue-os/bluefin/pull/1483. As a workaround I fixed it by accepting all signatures in /etc/containers/policy.json:

/etc/containers🔒 
❯ diff policy.json.bak policy.json -C 2
*** policy.json.bak 2024-08-22 09:46:37.796002073 +0200
--- policy.json 2024-08-22 09:46:53.903880061 +0200
***************
*** 32,40 ****
              "ghcr.io/ublue-os": [
                  {
!                     "type": "sigstoreSigned",
!                     "keyPath": "/usr/etc/pki/containers/ublue-os.pub",
!                     "signedIdentity": {
!                         "type": "matchRepository"
!                     }
                  }
              ],
--- 32,36 ----
              "ghcr.io/ublue-os": [
                  {
!                     "type": "insecureAcceptAnything"
                  }
              ],

Mortifier13 commented 2 months ago

Sounds like a different issue from the one I'm having. The issue I'm having is occurring on 2 independent systems installed separately and there's no errors when rpm-ostree update is run.

m2Giles commented 2 months ago

@lion7 your error is indeed the one that you linked. See the pinned issue to get the updated cosign public key instead of doing insecureAcceptAnything.

@Mortifier13 is this still occuring? It sounds like the deployment is failing; however, its confusing since you said an update works after a rollback. The only time I've had a similar issue was when I had specifically modified the root fs outside of an ostree update and then removed a package dependent on those changes.

Mortifier13 commented 2 months ago

@lion7 your error is indeed the one that you linked. See the pinned issue to get the updated cosign public key instead of doing insecureAcceptAnything.

@Mortifier13 is this still occuring? It sounds like the deployment is failing; however, its confusing since you said an update works after a rollback. The only time I've had a similar issue was when I had specifically modified the root fs outside of an ostree update and then removed a package dependent on those changes.

I think my issue is occurring due to a lack of free space on /root, it was almost full when I had a few pinned deployments so I removed some of them so I'm down to 3 total deployments in /boot/ostree/. My /boot is 1GB and with only 3 now instead of 5 it's still using about 750 megs. It worked on a manual update last night so I'm testing it over the next few days to see but there haven't been many daily updates rolling through the nvidia branch lately. If it seems to work on updating I'd like to increase the size of my /boot partition since I like to always keep a few pinned deployments on the latest branch.

ublue-os / bluefin