search

fedora-silverblue / issue-tracker

Fedora Silverblue issue tracker
https://fedoraproject.org/atomic-desktops/silverblue/
123 stars 3 forks source link

No updates since 39.20231205.0 #516

Closed chrisawi closed 11 months ago

chrisawi commented 11 months ago

The composes seem to be finishing correctly, but rpm-ostree upgrade isn't finding them. Notably, this means the fix for https://github.com/containers/podman/issues/20872 hasn't gone out yet.

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-39-updates-20231208.0/logs/x86_64/Everything/ostree-1/create-ostree-repo.log

Side note:

NOTICE: Running rpm-ostree compose tree without --unified-core is deprecated. Please add --unified-core to the command line and ensure your content works with it. For more information, see https://github.com/coreos/rpm-ostree/issues/729

Is that expected? I thought F39 was using unified core.

walking-octopus commented 11 months ago

Can reproduce on Kinoite.

stas@thinkpad-t14 ~> sudo rpm-ostree upgrade
[sudo] password for stas: 
note: automatic updates (stage) are enabled
2 metadata, 0 content objects fetched; 788 B transferred in 3 seconds; 0 bytes content written
Checking out tree 694f995... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Updating metadata for 'updates'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-09T02:29:32Z solvables: 15530
rpm-md repo 'fedora' (cached); generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'updates-archive'; generated: 2023-12-09T02:55:17Z solvables: 15912
Resolving dependencies... done
No upgrade available.
stas@thinkpad-t14 ~> rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 15h ago
Deployments:
● fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

  fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

  fedora:fedora/38/x86_64/kinoite
                  Version: 38.20231123.0 (2023-11-23T02:20:11Z)
               BaseCommit: 9b3d24e0139e07a6f334af5565761393b0a4d2b310a4bcfee77c2e0f5c4fc6d8
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: 'gcc-c++' automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop tmux xeyes
                   Pinned: yes

AvailableUpdate:
           Diff: 6 upgraded
gerryxiao commented 11 months ago

same here: gerryxiao@fedora ~> toolbox Error: failed to get the Podman version gerryxiao@fedora /> bat /etc/os-release 1 │ NAME="Fedora Linux" 2 │ VERSION="39.20231205.0 (Sericea)" 3 │ ID=fedora 4 │ VERSION_ID=39 5 │ VERSION_CODENAME="" 6 │ PLATFORM_ID="platform:f39" 7 │ PRETTY_NAME="Fedora Linux 39.20231205.0 (Sericea)" 8 │ ANSI_COLOR="0;38;2;60;110;180" 9 │ LOGO=fedora-logo-icon 10 │ CPE_NAME="cpe:/o:fedoraproject:fedora:39" 11 │ DEFAULT_HOSTNAME="fedora" 12 │ HOME_URL="https://fedoraproject.org/sericea/" 13 │ DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-sericea/" 14 │ SUPPORT_URL="https://ask.fedoraproject.org/" 15 │ BUG_REPORT_URL="https://gitlab.com/fedora/sigs/sway/SIG/-/issues" 16 │ REDHAT_BUGZILLA_PRODUCT="Fedora" 17 │ REDHAT_BUGZILLA_PRODUCT_VERSION=39 18 │ REDHAT_SUPPORT_PRODUCT="Fedora" 19 │ REDHAT_SUPPORT_PRODUCT_VERSION=39 20 │ SUPPORT_END=2024-11-12 21 │ VARIANT="Sericea" 22 │ VARIANT_ID=sericea 23 │ OSTREE_VERSION='39.20231205.0'

travier commented 11 months ago

This is indeed super weird. We'll need to file an infra issue.

RobusTetus commented 11 months ago

This is probably unrelated, although I am experiencing same issue, but on top of that GNOME Software reports glibc-langpack-cs can be updated. After installing and rebooting, it is still there as if it did not get installed.

travier commented 11 months ago

Filed: https://pagure.io/releng/issue/11827

fizzyizzy05 commented 11 months ago

This is probably unrelated, although I am experiencing same issue, but on top of that GNOME Software reports glibc-langpack-cs can be updated. After installing and rebooting, it is still there as if it did not get installed.

I'm having the same thing with glibc-langpack-en

GNOME software shows there's an available update, and appears to be installing it and offers to reboot, but it does nothing and the update is still there on reboot. Running rpm-ostree upgrade --preview shows there's an upgrade available to 2.38-14, but running rpm-ostree upgrade (no --preview or --check) shows it's fine.

chrisawi commented 11 months ago

Running rpm-ostree upgrade --preview shows there's an upgrade available to 2.38-14, but running rpm-ostree upgrade (no --preview or --check) shows it's fine.

This is a known issue in rpm-ostree: https://github.com/coreos/rpm-ostree/issues/1579

It pops up whenever base image updates are interrupted.

See also: https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1728

bitestring commented 11 months ago

Can confirm. Two of my Fedora Silverblue instances stopped receiving updates since version 39.20231205.0.

Also rpm-ostree is constantly busy

╰─>$ rpm-ostree status
State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Transaction: refresh-md
  Initiator: caller :1.72

doing nothing. Even rpm-ostree cancel does not work.

travier commented 11 months ago

Three options for workarounds until this is resolved:

travier commented 11 months ago

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

rscm commented 11 months ago

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

not yet, I tried updating and I'm still on the same BaseCommit

also this is not the first time it happens #475 

❯ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/39/x86_64/silverblue
                  Version: 39.20231205.0 (2023-12-05T00:55:28Z)
               BaseCommit: ae1afb462f69a9170bdfa24ef9fc3059b2366480c4fbcfac2a15c0fc65487d8a
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
      RemovedBasePackages: firefox firefox-langpacks 120.0.1-1.fc39 gnome-software gnome-software-rpm-ostree 45.1-1.fc39
          LayeredPackages: akmod-nvidia android-tools distrobox gnome-tweaks kernel-tools pipewire-codec-aptx printer-driver-brlaser rpmfusion-free-release rpmfusion-nonfree-release starship steam-devices tailscale terminator xboxdrv
                           xorg-x11-drv-nvidia

~

Can confirm. Two of my Fedora Silverblue instances stopped receiving updates since version 39.20231205.0.

Also rpm-ostree is constantly busy

╰─>$ rpm-ostree status
State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Transaction: refresh-md
  Initiator: caller :1.72

doing nothing. Even rpm-ostree cancel does not work.

that happened to me, at least in my case was gnome-software monopolizing the process (in a bad way)

there are two solutions/workarounds for that:

moljac024 commented 11 months ago

I am still having this issue:

❯ rpm-ostree update --preview
1 metadata, 0 content objects fetched; 592 B transferred in 0 seconds; 0 bytes content written
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora rpmfusion-nonfree-nvidia-driver google-chrome rpmfusion-nonfree-steam phracek-PyCharm updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'rpmfusion-nonfree-nvidia-driver'... done
Updating metadata for 'google-chrome'... done
Updating metadata for 'rpmfusion-nonfree-steam'... done
Updating metadata for 'phracek-PyCharm'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-13T01:20:39Z solvables: 15915
rpm-md repo 'fedora'; generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'rpmfusion-nonfree-nvidia-driver'; generated: 2023-11-30T17:20:04Z solvables: 29
rpm-md repo 'google-chrome'; generated: 2023-12-12T20:06:21Z solvables: 3
rpm-md repo 'rpmfusion-nonfree-steam'; generated: 2023-08-10T16:27:35Z solvables: 2
rpm-md repo 'phracek-PyCharm'; generated: 2023-08-10T15:35:19Z solvables: 5
rpm-md repo 'updates-archive'; generated: 2023-12-13T01:38:01Z solvables: 16689
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
AvailableUpdate:
       Upgraded: glibc-langpack-en 2.38-11.fc39 -> 2.38-14.fc39
                 gnutls-dane 3.8.2-1.fc39 -> 3.8.2-2.fc39
                 gnutls-utils 3.8.2-1.fc39 -> 3.8.2-2.fc39
❯ rpm-ostree update
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree ae1afb4... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora rpmfusion-nonfree-nvidia-driver google-chrome rpmfusion-nonfree-steam phracek-PyCharm updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'rpmfusion-nonfree-nvidia-driver'... done
Updating metadata for 'google-chrome'... done
Updating metadata for 'rpmfusion-nonfree-steam'... done
Updating metadata for 'phracek-PyCharm'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-13T01:20:39Z solvables: 15915
rpm-md repo 'fedora'; generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'rpmfusion-nonfree-nvidia-driver'; generated: 2023-11-30T17:20:04Z solvables: 29
rpm-md repo 'google-chrome'; generated: 2023-12-12T20:06:21Z solvables: 3
rpm-md repo 'rpmfusion-nonfree-steam'; generated: 2023-08-10T16:27:35Z solvables: 2
rpm-md repo 'phracek-PyCharm'; generated: 2023-08-10T15:35:19Z solvables: 5
rpm-md repo 'updates-archive'; generated: 2023-12-13T01:38:01Z solvables: 16689
Resolving dependencies... done
No upgrade available.
RobusTetus commented 11 months ago

I don't know if the today's compose happened yet. But I can confirm, still nothing over here too. :cry:

Mershl commented 11 months ago

I don't know if the today's compose happened yet. But I can confirm, still nothing over here too. 😢

You can query for the latest available image on the repo using ostree remote summary fedora | grep -A3 "fedora/39/x86_64/silverblue"

Version (ostree.commit.version): 39.20231205.0 -> 5th December 2023

travier commented 11 months ago

Building with unified core is fixed. The builds are still not pushed to the repo. Still needs more investigation.

RobusTetus commented 11 months ago

Building with unified core is fixed. The builds are still not pushed to the repo. Still needs more investigation.

My bad I thought that it will get built today.

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

I misunderstood your message.

SnuggleCovenant commented 11 months ago

Three options for workarounds until this is resolved:

* Switch to the unofficial container images (they are updated daily):
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/silverblue:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/kinoite:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/sericea:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/onyx:39
* Replace podman by a working version:
# 4.8.1 from stable repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-e8cfbfbdd9
# or 4.8.2 from testing repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-68ad6ada3b
* If you don't mind losing all containers, images, volumes:
$ podman system reset

I tried the second option.

$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-e8cfbfbdd9
$ systemctl reboot
$ rpm-ostree cleanup -m
$ rpm-ostree upgrade
[...] No upgrade available.
$ ostree remote summary fedora | grep -A3 "fedora/39/x86_64/silverblue"
* fedora/39/x86_64/silverblue
Latest Commit (61,0 kB):
ae1afb462f69a9170bdfa24ef9fc3059b2366480c4fbcfac2a15c0fc65487d8a
Version (ostree.commit.version): 39.20231205.0

I also tried the first option which is rebasing on the latest nightly. ostree remote summary still shows the base from 5 dec 23 so that makes me think i might have to keep upgrading by rebasing regularly?

I think I'll switch to ublue since they are unaffected and signed.

I'm surprised how little attention this issue seems to be getting from what I can gather as a simple user.

travier commented 11 months ago

ublue images are based on the images listed in the first workaround.

The second and third options are not going to get you updates but only a working podman if you are affected by the issue from the 4.8.0 release: https://github.com/fedora-silverblue/issue-tracker/issues/513.

Froggy232 commented 11 months ago

Hi, Is there an ETA for when it will be solved? Thanks you

miabbott commented 11 months ago

Is there an ETA for when it will be solved?

No ETA at this time; you can follow progress in the Fedora ticket - https://pagure.io/releng/issue/11827

sn4201 commented 11 months ago

I'm experiencing this same behaviour with the glibc update showing as always available but not receiving anything isnce 12-05-2023

So am I correct in understanding that we simply wait for the usual update mechanism to have a fix pushed remotely? Will it simply start working again or will there be some manual intervention required afterward?

Mershl commented 11 months ago

39.20231215.1 just popped up on the repo. Happy updating everybody.

rene-geile commented 11 months ago

Hi, successfully upgraded now with lots of Packages to 39.20231215.1 Previous attempt ~~19:20 CET failed ( online 1 Package SDL2 Update) so it seems directly related to the fix.

travier commented 11 months ago

Should be good for Fedora 39. Fedora 38 will get its update tomorrow.

RobusTetus commented 11 months ago

Should be good for Fedora 39. Fedora 38 will get its update tomorrow.

Can confirm it's all good for me on Silverblue. Successfully updated using the rpm-ostree update command. Thank you for your hard work. :+1:

travier commented 11 months ago

Should be fixed for F38 & F39 now. Closing.