[BUG] Updates tab: “Operating System Updates Unavailable: your operating system is no longer supported.”

[aral]

Describe the bug

When you open Software and go to the Updates tab, you see an error: “Operating System Updates Unavailable: your operating system is no longer supported. This means that it does not receive security updates. It is recommended that you update to a more recent version.”

To Reproduce

Please describe the steps needed to reproduce the bug:

1. Install Fedora Silverblue 36 beta
2. Wait for the notification that you have updates available and click on it.
3. Notice the error message above.

Expected behavior

The updates tab should not display an error message. You should be able to see the updates that were mentioned in the system notification that got you there.

Screenshots

OS version:

● fedora:fedora/36/x86_64/silverblue
                   Version: 36_Beta.1.4 (2022-03-22T03:27:52Z)
                    Commit: 99b8bb190924784dd5a3076f830a4bf70f21a34437b3762abba8ca416e9c9798

Additional context

I’d just installed the operating system and this was one of my first experiences.

[leb4r]

I tried out version 36.20220514.0 and it works very well!

@ALL developers: thanks you very much!!

Only one problem remains:

This may be unrelated, but I've also noticed a lot of legacy apps are still around. For example, the old screenshots app is still around (along with the new one), and a lot of the apps that were updated with the new interface are still using the legacy interface (e.g. gedit).

Does somebody knows how to solve this? Is there a separate bug report for this?

I think this is outlined in https://github.com/fedora-silverblue/issue-tracker/issues/273, specifically this statement in https://github.com/fedora-silverblue/issue-tracker/issues/273#issuecomment-1123658937:

Yes, that's expected at this point. The work on F36 Flatpak runtime is in progress and should be finished during next week. Then we will start to rebuild the applications against it and start to ship them to users.

---

For the apps that were replaced, you can simply uninstall those if you already have the replacements (e.g. gedit or screenshot tool).

[mks-h]

Just confirming, was able to update this evening, gnome-software is now functioning as I expect it to.

I tried out version 36.20220514.0 and it works very well!

Not the case for me, on version Fedora Linux 36.20220516.0 (Silverblue). Issue remains.

[mcrha]

Could you verify what package versions you've installed, please?

   $ rpm -q rpm-ostree libdnf libsolv gnome-software

[MateusRodCosta]

Just confirming, was able to update this evening, gnome-software is now functioning as I expect it to.

I tried out version 36.20220514.0 and it works very well!

Not the case for me, on version Fedora Linux 36.20220516.0 (Silverblue). Issue remains.

@delight-aug Have you tried following the steps at https://github.com/fedora-silverblue/issue-tracker/issues/257#issuecomment-1126392386 ?

You most likely didn't reset the libdnf override, so you are still having incompatible libdnf versions.

[Verhoeckx]

I think this is outlined in #273, specifically this statement in #273 (comment):

Ah, good to hear to some people are working on it! I think I will wait a few weeks until I update my system.

Maybe Fedora Silverblue 36 was released to early? Next time they could better wait a few weeks until all problems are fixed. I also noted a Podman bug but I will debug that after I have upgraded.

[mks-h]

@mcrha, sure, look:

rpm-ostree-2022.9-1.fc36.x86_64
libdnf-0.67.0-2.fc36.x86_64
libsolv-0.7.22-1.fc36.x86_64
gnome-software-42.1-1.fc36.x86_64

@MateusRodCosta, you're right, I didn't follow these steps, because I haven't been downgrading libdnf. I did now, by grabbing its package libdnf-0.66.0-1.fc36.x86_64 from the koji link you provided — it still doesn't work. Then, I've reset it back, obviously, with no luck either.

[mks-h]

Maybe Fedora Silverblue 36 was released too early?

Not just early, it feels like they've done no testing. To be honest, I didn't expect issues like that from a system that advertises as "aiming to be extremely stable and reliable".

[tpopela]

@delight-aug the Silverblue 36 as it was released was working fine, I was using it without any problems. The problem was that the first update that went out completely broke it - https://bodhi.fedoraproject.org/updates/FEDORA-2022-ea31b1d9c5. It was tested by several people (probably none of them was using Fedora Silverblue), got enough karma and was pushed into stable. There sadly isn't anything (around the processes or Fedora release infrastructure that could prevent it) and that it happened just a day after the Silverblue 36 release is just a coincidence.

[MateusRodCosta]

@mcrha, sure, look:

rpm-ostree-2022.9-1.fc36.x86_64
libdnf-0.67.0-2.fc36.x86_64
libsolv-0.7.22-1.fc36.x86_64
gnome-software-42.1-1.fc36.x86_64

@MateusRodCosta, you're right, I didn't follow these steps, because I haven't been downgrading libdnf. I did now, by grabbing its package libdnf-0.66.0-1.fc36.x86_64 from the koji link you provided — it still doesn't work. Then, I've reset it back, obviously, with no luck either.

@delight-aug

Hi, I think you are having the other part of the bug, the one I didn't personally hit but predicted would likely happen where caches conflict, you might need also to clean completely rpm-ostree cache and recreate it (so remember that there was apparently a change in how the cache formatting or versioning, something like that).

From my other comment on https://bugzilla.redhat.com/show_bug.cgi?id=2083715#c24 , follow those steps (with the current libdnf, not the downgraded one):

Run gnome-sotware --quit (close Gnome Software so it doesn't interfere, including stopping it from running in the background), rpm-ostree cleanup --repomd (should fix any rpm-ostree cache conflicts between the two libdnf versions, by removing everything) and rpm-ostree refresh-md (gnome-software can't create /var/cache/rpm-ostree/repomd/ and /var/cache/rpm-ostree/solv/ if they don't exist, they were deleted when cleaning the repo cache and are recreated when refreshing it) and then run GNOME Software as normal.

If even then you still have problems, it's likely due to Gnome Software's own cache, run the same steps as above, but just after running gnome-software --quit, delete the folder ~/.cache/gnome-software (Enable hidden files in Nautilus if trying via GUI, so you can find the hidden .cache folder, Ctrl+H or the "Show hidden files"option in the hamburguer menu should allow you to enable that), then run the two other rpm-ostree commands in order.

After that things should work.

If even after that it doesn't work, then likely the fact that gnome-software tries to create a /var/run/dnf-metadata.lock.<random string> and fails might be to blame.

This could be it because of the above libdnf problem, could be because it's trying to when it shouldn't and it's failing, could be because while it should be trying to create it, something else prevents it from.

On my system /var/run is a symlink to /run:

[mateusrc@centauro run]$ ls -al /var/run
lrwxrwxrwx. 1 root root 6 mar  2 16:47 /var/run -> ../run

/run is mounted like this:

[mateusrc@centauro run]$ mount | grep /run
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=3237204k,nr_inodes=819200,mode=755,inode64)
<snip>

If everything looks alright then... Then, no idea, maybe SELinux? Can you try investigating logs to see if something hints at the problem?

I do know that there is a common issue on upgrading to Fedora 36 due to SELinux, which can been seen here. Although the steps to fix seem to be exclusive to non-rpm-ostree systems...

Maybe Fedora Silverblue 36 was released too early?

Not just early, it feels like they've done no testing. To be honest, I didn't expect issues like that from a system that advertises as "aiming to be extremely stable and reliable".

I was in the Beta version (the same stable ref is used for that) before release (basically the May 08 image) and it was working there, the case here seems to be that they decided to push out the new libdnf in the May 10 image for the day of release. So, as far as I'm aware it didn't get to Beta AND LIKELY WASN'T EVEN IN THE RC5, and might have been pushed from testing to stable on release day.

The way I can see it explained is that it got through testing -> stable because nobody noticed it, which either means that nobody updated to that on Silverblue running testing or that no one that runs Silverblue on testing used GNOME Software with the update applied.

And, at least for me, it seems that it would make a lot of sense that users who run Fedora Silverblue testing would prefer to use rpm-ostree CLI instead of Gnome Software, explaining why nobody noticed. (i.e. Nobody noticed before it hit stable because only when it hit stable that came the users actually using GNOME Software with the combination rpm-ostree + newer libdnf).

Since the actual point in which the Fedora 36 was given a GO was with the 1.5 RC, which was from May 04:

commit 16ec3d1166e5281e2deb1beee62e037777bf2d013f82165adf71b7885c5a53f0
Parent:  e35b05b5ae554bb1385a5cb3dbcbfa3ece54d05658fec247853abd5161363466
ContentChecksum:  0bd7a6891ffdd5277b746408103a0e145580cb9df7af6738ef8a27f6c77e4b21
Date:  2022-05-04 18:42:06 +0000
Version: 36.1.5
(no subject)

And the libdnf version there:

[mateusrc@centauro run]$ rpm-ostree db list 16ec3d1166e5281e2deb1beee62e037777bf2d013f82165adf71b7885c5a53f0 | grep libdnf
 libdnf-0.66.0-1.fc36.x86_64

And, since the libdnf package was introduced between the 36.20220508.0 (commit dd4ac38b4030e0192777eef2c243f1b2a777f6d6526fd52b84f0a2ec2984b6bb and 36.20220510.0 (commit 8e68c41bf30c73f76be8da7d1fd3be18d63ce33379ecf5c4267a5b42c20d9d74) versions, this was much more of a "Day 1 update breaks functionality" than a "They didn't test enough before release".

(As usual, rpm-ostree db diff dd4ac38b4030e0192777eef2c243f1b2a777f6d6526fd52b84f0a2ec2984b6bb 8e68c41bf30c73f76be8da7d1fd3be18d63ce33379ecf5c4267a5b42c20d9d74 to compare them)

This would have been prevented instead by some sort of requirement for pushing to stable for the tools that install software, libdnf or dnf (or whatever other is the actual frontend) and rpm-ostree (even flatpak if wanting to expanding from package managers), to not break whatever are the main GUI applications for Software updates.

Something of the sort should be implemented to prevent something like this from happening again, but this time it was almost unavoidable, maybe only really by luck. For example, if Fedora 36 was released one week earlier and there were still no Silverblue 36 testing users to run Gnome Software, it would still be pushed to stable, same thing would happen except it wouldn't be blamed on the release itself, just a bad update. If Fedora 36 was delayed one more week, the update would be in the stable repos for RC creation and it most likely would have been caught in the RC. Just unfortunate timing.

Edit: Yeah, basically @tpopela sent above while I was still typing this (lol), just coincidence. The process around giving karma to such critical tools should be improved a bit though.

[o-kotb]

So, as far as I'm aware it didn't get to Beta AND LIKELY WASN'T EVEN IN THE RC5, and might have been pushed from testing to stable on release day.

I can confirm, when I installed the final release (it wasn't yet announced but rc5 was indeed the final release), gnome software was working fine and only broke after the update. So the actual release is fine afaik. I think it was just weird timing.

[mks-h]

@tpopela, @MateusRodCosta, thanks for the detailed responses — I wasn't expecting that to my rant.

As for the cache refreshing — it didn't help, the problem remains. This is what my /var/run looks like:

[mks@fedora ~]$ ls -al /var/run
lrwxrwxrwx. 1 root root 6 May  5 22:58 /var/run -> ../run
[mks@fedora ~]$ mount | grep /run
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=1602624k,nr_inodes=819200,mode=755,inode64)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801308k,nr_inodes=200327,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Then, no idea, maybe SELinux?

I've run the commands listed at the end of that post, and It seems I'm not affected.

So, logically, if we have the same base OS versions, the reason for the bug is either in my overlays or something inside var. My overlays probably aren't an issue:

RemovedBasePackages: gnome-terminal-nautilus gnome-terminal 3.44.0-1.fc36
    LayeredPackages: akmod-nvidia cloudflare-warp ffmpeg gnome-console gnome-console-nautilus gnome-text-editor gnome-tweaks syncthing xorg-x11-drv-nvidia
                     xorg-x11-drv-nvidia-cuda xorg-x11-drv-nvidia-cuda-libs
      LocalPackages: cloudflare-release-3.0.0-el8.noarch rpmfusion-free-release-36-1.noarch rpmfusion-nonfree-release-36-1.noarch

---

I was in the Beta version (the same stable ref is used for that) before release (basically the May 08 image) and it was working there

Same here

AND LIKELY WASN'T EVEN IN THE RC5

Just... damn

[tpopela]

@delight-aug then please refrain from using rants at all in this forum - they're not helpful at all and just bring down the mood.

[Verhoeckx]

@delight-aug then please refrain from using rants at all in this forum - they're not helpful at all and just bring down the mood.

Maybe it's because I brought up the issue?

I don't want to bring down the mood, but I think it would have been better if they had delayed the release until all GNOME application would have been upgraded to the latest release (see issue #273). After all that's the biggest change of Fedora 36.

Having said that: I'm very happy with Fedora Silverblue overal :smiley: l!

[rowbawts]

Run gnome-sotware --quit (close Gnome Software so it doesn't interfere, including stopping it from running in the background), rpm-ostree cleanup --repomd (should fix any rpm-ostree cache conflicts between the two libdnf versions, by removing everything) and rpm-ostree refresh-md (gnome-software can't create /var/cache/rpm-ostree/repomd/ and /var/cache/rpm-ostree/solv/ if they don't exist, they were deleted when cleaning the repo cache and are recreated when refreshing it) and then run GNOME Software as normal.

Following this sequence of commands fixed it for me so thanks for that, I have no layered packages or overrides so that is probably why this worked for me pretty painlessly.

[mks-h]

@tpopela, refrain from ranting about my rant, then 🙃

[tpopela]

Maybe it's because I brought up the issue?

It's not that important who brought up the issue.

I don't want to bring down the mood, but I think it would have been better if they

Who's they? I'm a kind of representative of the Fedora project here (as a Silverblue's Product Owner).

had delayed the release until all GNOME application would have been upgraded to the latest release (see issue #273).

Fedora SIlverblue isn't a blocking release deliverable so the whole Fedora Project won't block Fedora 36 release on any of the Silverblue issues (I suspect apart from those that could prevent the successful booting and so on). And shipping with older applications isn't a reason to block the release for additional weeks (I'm already working for nearly 3 weeks on creating the Fedora 36 Flatpak runtime and even tough I'm nearly done, there's still some work that needs to be done).

After all that's the biggest change of Fedora 36.

No it isn't and that's what Silverblue is trying to pioneer by decoupling the applications from the base operating system - it's the combination of both and with applications decoupled we can actually deliver one part of it (newer applications) into older Silverblue releases. This cycle it's a different though due to capacity problems (that you're aware of if you read the issue #273)

[dustymabe]

This is a great time to chime in and say that if you like Silverblue we need help making it even better. Get involved and help steer the direction of it in the future.

[thomasazar]

I am also having this problem, but only with the ProtonVPN repo installed.

I discovered this problem on my newly rebased Fedora Silverblue 36, and after removing the Tailscale repo from /etc/yum.repos.d, then running the commands to cleanup/refresh the repos, GNOME Software is working as expected.

[AdamIsrael]

I discovered this problem on my newly rebased Fedora Silverblue 36, and after removing the Tailscale repo from /etc/yum.repos.d, then running the commands to cleanup/refresh the repos, GNOME Software is working as expected.

Removing the Tailscale repo also solved this for me, too.

Does anyone know why the Tailscale repo breaks GNOME Software, though? I'd like to continue receiving updates to Tailscale, so running without the repo installed is problematic.

[bobslept]

Does anyone know why the Tailscale repo breaks GNOME Software, though? I'd like to continue receiving updates to Tailscale, so running without the repo installed is problematic.

Could it be something with the repo not being ready for F36? https://pkgs.tailscale.com/stable/fedora/36/tailscale.repo returns a 404. While the same with 35 in the url returns a repo file. Not a user myself, so I'm only guessing.

[mcrha]

Could it be something with the repo not being ready for F36?

That could be the reason. My wild guess is:

• rpm-ostree updates the repos cache and fails with the missing f36 repo
• the dnf called from the gnome-software sees missing repo cache, thus it tries to create it, but it doesn't have permissions to write files into the directory, thus it fails and errors out.

[AdamIsrael]

Thanks, @bobslept and @mcrha , that reasoning makes sense.

[Verhoeckx]

Fedora SIlverblue isn't a blocking release deliverable so the whole Fedora Project won't block Fedora 36 release on any of the Silverblue issues (I suspect apart from those that could prevent the successful booting and so on). And shipping with older applications isn't a reason to block the release for additional weeks (I'm already working for nearly 3 weeks on creating the Fedora 36 Flatpak runtime and even tough I'm nearly done, there's still some work that needs to be done).

No it isn't and that's what Silverblue is trying to pioneer by decoupling the applications from the base operating system - it's the combination of both and with applications decoupled we can actually deliver one part of it (newer applications) into older Silverblue releases. This cycle it's a different though due to capacity problems (that you're aware of if you read the issue #273)

Another solution would have been to mentioned the delayed GTK4 apps in the release notes (on the website silverblue.fedoraproject.org)? Something like: this release won't contain the new GTK4 applications of the GNOME desktop but we will update them in the coming weeks (after building the new Flatpak runtime environment).

Having said that: thanks for your work on Fedora Silverblue!!

I recently tried to promote Fedora Silverblue a bit by writing a short tutorial about how you can run Laravel (a PHP framework) with Podman on Fedora Silverblue. You can find the short tutorial here.

[mks-h]

Could it be something with the repo not being ready for F36?

Can confirm, for me it was Cloudflare's repo, which is actually for CentOS 8. Removed the repo, and I finally have working Software.

[AdamIsrael]

Could it be something with the repo not being ready for F36?

Can confirm, for me it was Cloudflare's repo, which is actually for CentOS 8. Removed the repo, and I finally have working Software.

I can't confirm yet, but it seems that the issue, at least for me and Tailscale, is related to gpg signature verification. Tailscale added a repo for F36 last night, so I re-added the repo and gnome-software is once again broken. This time, I've captured some logs that point in a new direction.

02:13:33:0820 librepo lr_handle_perform: Downloading/Locating yum repo
02:13:33:0820 librepo lr_yum_use_local: Locating repo..
02:13:33:0820 librepo lr_yum_use_local_load_base: Parsing repomd.xml
02:13:33:0846 librepo lr_gpg_check_signature_fd: signature verify error (no signatures)
02:13:33:0846 librepo lr_yum_use_local_load_base: repomd.xml GPG signature verification failed: Signature verify error - no signatures
02:13:33:0846 libdnf failed to check, attempting update: repodata tailscale-stable was not complete: repomd.xml GPG signature verification failed: Signature verify error - no signatures
02:13:33:0846 libdnf setting keyfile data for tailscale-stable
02:13:33:0846 libdnf Failed to remove /var/cache/rpm-ostree/repomd/tailscale-stable-36-x86_64.tmp: cannot open directory /var/cache/rpm-ostree/repomd/tailscale-stable-36-x86_64.tmp: Error opening directory “/var/cache/rpm-ostree/repomd/tailscale-stable-36-x86_64.tmp”: No such file or directory
02:13:33:0851 Gs  Setting I/O priority of thread 0x5562c7cf7180 to IDLE, 7
02:13:33:0851 Gs  updates-shell: failed to get updates: failed to obtain lock 'metadata': Failed to create file “/var/run/dnf-metadata.lock.5XKIM1”: Permission denied

The repo does have a gpg key:

[tailscale-stable]
name=Tailscale stable
baseurl=https://pkgs.tailscale.com/stable/fedora/36/$basearch
enabled=1
type=rpm
repo_gpgcheck=1
gpgcheck=0
gpgkey=https://pkgs.tailscale.com/stable/fedora/36/repo.gpg

And if I disable repo_gpgcheck, gnome-software works as expected. I'm starting to suspect this is a larger Silverblue issue; when I check the other repos, every single one has repo_gpgcheck disabled.

if you're still having the problem, try running grep "repo_gpgcheck=1" /etc/yum.repos.d/*. If you get any hits, try disabling repo_gpgcheck and try software-center again.

[MateusRodCosta]

So, while the libdnf version mismatch between rpm-ostree and gnome-software issue is fixed, it seems some people are hitting a new issue where, due to incompatible repos, Gnome Software breaks again, and apparently rpm-ostree is also affected.

Maybe it's actually a libdnf issue and you guys should report it there and also try to reproduce in Workstation? (likely dnf and gnome-software there are also affected).

[rscm]

grep "repo_gpgcheck=1" /etc/yum.repos.d/*

thanks, I found the same on the repo file

but also found that gpgcheck was disabled, maybe they made a mistake so I enabled it and now also everything works

[tpopela]

Closing this issue as it has been resolved. If anyone is experiencing new issues as @MateusRodCosta is suggesting, please open a new one.

[1player]

So, what's actually broken, libdnf, gnome-software or the repos using repo_gpgcheck=1 ? I'd like to open bug reports where appropriate so this doesn't happen again.

EDIT: this needs to be reopened because the same repo file works on Workstation and not on Silverblue, so it's a bug, and quite annoying one to hit.

[MateusRodCosta]

So, what's actually broken, libdnf, gnome-software or the repos using repo_gpgcheck=1 ? I'd like to open bug reports where appropriate so this doesn't happen again.

EDIT: this needs to be reopened because the same repo file works on Workstation and not on Silverblue, so it's a bug, and quite annoying one to hit.

Hi, it seems from your edit you did follow my suggestion from a while ago:

Maybe it's actually a libdnf issue and you guys should report it there and also try to reproduce in Workstation? (likely dnf and gnome-software there are also affected).

(Although you could likely have just used a Fedora 36 toolbox for that)

Even then, the bug you are now hitting is still not this bug, as this one is about mismatched libdnf versions between g-s and rpm-ostree generating incompatible cached repo data which breaks g-s.

But probably it's something likely related to rpm-ostree, however the only issue that seems related to this I can find there is https://github.com/coreos/rpm-ostree/issues/2508. It seems that support for repo_gpgcheck is already available for rpm-ostree though, but might not be enabled for Silverblue. It might be something more complex than that though.

EDIT: Since it seems to only break Gnome Software and rpm-ostree from command line works fine, I would guess it's a bug somewhere else in the integration between rpm-ostree and gnome-software, either via libdnf or gnome-software-rpm-ostree.

[1player]

Even then, the bug you are now hitting is still not this bug

In that case I've opened #295 to track the issue about repo_gpgcheck breaking GNOME Software.

[mcrha]

GNOME Software under Silverblue uses rpm-ostree and direct calls to libdnf, while GNOME Software under Workstation (and other spins) uses PackageKit, with no libdnf direct use. That explains the difference why the bug exhibits under Silverblue, but not under other spins.

[cgwalters]

https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1793