`dnf5 list` shows unknown repo for packages installed with DNF 4

[evan-goode]

dnf5 list shows the repo as <unknown> for packages installed with DNF 4, and vice versa: dnf4 list shows the repo as @System for packages installed with DNF 5.

On a fresh Fedora 39 container:

[root@2d825ae35a5f /]# dnf4 install dnf5 -y
[root@2d825ae35a5f /]# dnf5 install hello -y

[root@2d825ae35a5f /]# dnf4 list hello
Last metadata expiration check: 0:08:35 ago on Thu Apr  4 16:48:05 2024.
Installed Packages
hello.x86_64                             2.12.1-2.fc39                              @System

[root@2d825ae35a5f /]# dnf5 list hello
Updating and loading repositories:
Repositories loaded.
Installed packages
hello.x86_64 2.12.1-2.fc39 fedora

[root@2d825ae35a5f /]# dnf4 list --installed
Installed Packages
alternatives.x86_64                      1.25-1.fc39                @anaconda       
audit-libs.x86_64                        3.1.2-5.fc39               @koji-override-1
...

[root@2d825ae35a5f /]# dnf5 list --installed
Installed packages
alternatives.x86_64                      1.25-1.fc39                <unknown>
audit-libs.x86_64                        3.1.2-5.fc39               <unknown>
...

Reported by multiple users during the 2023-08-11 DNF 5 test day and the 2024-03-15 DNF 5 test day.

[j-mracek]

DNF and DNF5 does not share internal information from which repository a package was installed. DNF5 uploads those information from DNF4 once and then it keeps own independent copy.

The information about repository is restored in DNF5 after a package is modified by DNF5. But this is not true for package reason. When new package is installed by DNF4, DNF5 will not known a reason for this package.

I am really sorry but we cannot do much here, therefore I am closing the issue.

[jan-kolarik]

DNF and DNF5 does not share internal information from which repository a package was installed. DNF5 uploads those information from DNF4 once and then it keeps own independent copy.

The information about repository is restored in DNF5 after a package is modified by DNF5. But this is not true for package reason. When new package is installed by DNF4, DNF5 will not known a reason for this package.

I am really sorry but we cannot do much here, therefore I am closing the issue.

I thought we were discussing at least describing this behavior in our documentation. Or do you think even that would not make sense?

[jan-kolarik]

I tested the following use case I would expect to be working. I installed dnf5 on the VM system where only dnf4 was used before. When running dnf5 distro-sync as the first command after the dnf5 installation, I still get the <unknown> repositories for the packages being replaced in the transaction.

# dnf5 distro-sync
Updating and loading repositories:
Repositories loaded.
Package                                   Arch       Version                                    Repository                 Size
Upgrading:                                                                                                                     
 expat                                    x86_64     2.6.2-1.fc39                               updates               288.6 KiB
  replacing expat                         x86_64     2.6.0-1.fc39                               <unknown>             284.7 KiB
 libblockdev                              x86_64     3.1.1-1.fc39                               updates               358.5 KiB
  replacing libblockdev                   x86_64     3.1.0-1.fc39                               <unknown>             358.4 KiB
 libblockdev-crypto                       x86_64     3.1.1-1.fc39                               updates                63.6 KiB
  replacing libblockdev-crypto            x86_64     3.1.0-1.fc39                               <unknown>              63.6 KiB
 libblockdev-fs                           x86_64     3.1.1-1.fc39                               updates               112.6 KiB
  replacing libblockdev-fs                x86_64     3.1.0-1.fc39                               <unknown>             112.6 KiB
 libblockdev-loop                         x86_64     3.1.1-1.fc39                               updates                23.5 KiB
  replacing libblockdev-loop              x86_64     3.1.0-1.fc39                               <unknown>              23.5 KiB
 libblockdev-mdraid                       x86_64     3.1.1-1.fc39                               updates                35.7 KiB
  replacing libblockdev-mdraid            x86_64     3.1.0-1.fc39                               <unknown>              35.7 KiB
 libblockdev-nvme                         x86_64     3.1.1-1.fc39                               updates                47.4 KiB
  replacing libblockdev-nvme              x86_64     3.1.0-1.fc39                               <unknown>              47.4 KiB
 libblockdev-part                         x86_64     3.1.1-1.fc39                               updates                43.4 KiB
  replacing libblockdev-part              x86_64     3.1.0-1.fc39                               <unknown>              43.4 KiB
 libblockdev-swap                         x86_64     3.1.1-1.fc39                               updates                23.6 KiB
  replacing libblockdev-swap              x86_64     3.1.0-1.fc39                               <unknown>              23.6 KiB
 libblockdev-utils                        x86_64     3.1.1-1.fc39                               updates                43.6 KiB
  replacing libblockdev-utils             x86_64     3.1.0-1.fc39                               <unknown>              43.6 KiB
 libeconf                                 x86_64     0.5.2-2.fc39                               updates                55.9 KiB
  replacing libeconf                      x86_64     0.5.2-1.fc39                               <unknown>              55.9 KiB
 librepo                                  x86_64     1.17.1-1.fc39                              updates               244.2 KiB
   replacing librepo                      x86_64     1.17.0-1.fc39                              <unknown>             244.9 KiB

Looking at the dnf history info 1 output, I see all these packages coming from the inital installation:

... snip ...

    Install librepo-1.17.0-1.fc39.x86_64                              @anaconda

... snip ...

Therefore it seems to me we could improve the situation ...

[jan-kolarik]

I tried running distro-sync again after the history info 1 and now I get the repositories correctly:

# dnf5 distro-sync
Updating and loading repositories:
Repositories loaded.
Package                                   Arch       Version                                    Repository                 Size
Upgrading:                                                                                                                     
 expat                                    x86_64     2.6.2-1.fc39                               updates               288.6 KiB
  replacing expat                         x86_64     2.6.0-1.fc39                               updates               284.7 KiB
 libblockdev                              x86_64     3.1.1-1.fc39                               updates               358.5 KiB
  replacing libblockdev                   x86_64     3.1.0-1.fc39                               updates               358.4 KiB
 libblockdev-crypto                       x86_64     3.1.1-1.fc39                               updates                63.6 KiB
  replacing libblockdev-crypto            x86_64     3.1.0-1.fc39                               updates                63.6 KiB
 libblockdev-fs                           x86_64     3.1.1-1.fc39                               updates               112.6 KiB
  replacing libblockdev-fs                x86_64     3.1.0-1.fc39                               updates               112.6 KiB
 libblockdev-loop                         x86_64     3.1.1-1.fc39                               updates                23.5 KiB
  replacing libblockdev-loop              x86_64     3.1.0-1.fc39                               updates                23.5 KiB
 libblockdev-mdraid                       x86_64     3.1.1-1.fc39                               updates                35.7 KiB
  replacing libblockdev-mdraid            x86_64     3.1.0-1.fc39                               updates                35.7 KiB
 libblockdev-nvme                         x86_64     3.1.1-1.fc39                               updates                47.4 KiB
  replacing libblockdev-nvme              x86_64     3.1.0-1.fc39                               updates                47.4 KiB
 libblockdev-part                         x86_64     3.1.1-1.fc39                               updates                43.4 KiB
  replacing libblockdev-part              x86_64     3.1.0-1.fc39                               updates                43.4 KiB
 libblockdev-swap                         x86_64     3.1.1-1.fc39                               updates                23.6 KiB
  replacing libblockdev-swap              x86_64     3.1.0-1.fc39                               updates                23.6 KiB
 libblockdev-utils                        x86_64     3.1.1-1.fc39                               updates                43.6 KiB
  replacing libblockdev-utils             x86_64     3.1.0-1.fc39                               updates                43.6 KiB
 libeconf                                 x86_64     0.5.2-2.fc39                               updates                55.9 KiB
  replacing libeconf                      x86_64     0.5.2-1.fc39                               anaconda               55.9 KiB
 librepo                                  x86_64     1.17.1-1.fc39                              updates               244.2 KiB
   replacing librepo                      x86_64     1.17.0-1.fc39                              anaconda              244.9 KiB

So it seems like there is at least an issue when the system state is loaded from dnf4. According to logs it was loaded only after calling the history info 1 for the first time ...

[j-mracek]

It looks like that there is more problems with it. We have a code that transfer data from DNF4, but it looks like that it does not transfer even reasons correctly. dnf repoquery --installed --qf "%{name} %{reason} %{from_repo}"

xz dependency anaconda
xz-libs dependency anaconda
yajl dependency anaconda
yelp group anaconda
yelp-libs dependency anaconda
yelp-xsl dependency anaconda
yum group anaconda
zchunk-libs dependency anaconda
zenity dependency anaconda
zeromq dependency anaconda
zfs-fuse dependency anaconda
zimg dependency anaconda
zip group anaconda
zlib dependency anaconda
zram-generator dependency anaconda
zram-generator-defaults group anaconda
zvbi dependency anaconda
zxing-cpp dependency anaconda

dnf5 repoquery --installed --qf "%{name} %{reason} %{from_repo}\n"

xz External User <unknown>
xz-libs External User <unknown>
yajl External User <unknown>
yelp External User <unknown>
yelp-libs External User <unknown>
yelp-xsl External User <unknown>
yum External User <unknown>
zchunk-libs External User <unknown>
zenity External User <unknown>
zeromq External User <unknown>
zfs-fuse External User <unknown>
zimg External User <unknown>
zip External User <unknown>
zlib External User <unknown>
zram-generator External User <unknown>
zram-generator-defaults External User <unknown>
zvbi External User <unknown>
zxing-cpp External User <unknown>

[j-mracek]

Package reasons cannot be recovered by next transaction therefore I consider the issue as more critical than in past.

[j-mracek]

Also the state of groups is not transferred correctly

dnf group list
[sudo] password for hrom: 
Sorry, try again.
[sudo] password for hrom: 
Last metadata expiration check: 0:13:41 ago on Thu 11 Apr 2024 01:59:02 PM CEST.
Available Environment Groups:
   Fedora Custom Operating System
   Minimal Install
   Fedora Server Edition
   Fedora Cloud Server
   KDE Plasma Workspaces
   Xfce Desktop
   Phosh Desktop
   LXDE Desktop
   LXQt Desktop
   Cinnamon Desktop
   MATE Desktop
   Sugar Desktop Environment
   Deepin Desktop
   Budgie Desktop
   Development and Creative Workstation
   Web Server
   Infrastructure Server
   Basic Desktop
   i3 desktop
   Sway Desktop
Installed Environment Groups:
   Fedora Workstation
Installed Groups:
   Container Management
   LibreOffice
Available Groups:
   3D Printing
   Administration Tools
   Audio Production
   Authoring and Publishing
   Budgie
   Budgie Desktop Applications
   C Development Tools and Libraries
   Cloud Infrastructure
   Cloud Management Tools
   Compiz
   D Development Tools and Libraries
   Design Suite
   Development Tools
   Domain Membership
   Editors
   Educational Software
   Electronic Lab
   Engineering and Scientific
   FreeIPA Server
   Games and Entertainment
   Headless Management
   MATE Applications
   Milkymist
   Network Servers
   Neuron Modelling Simulators
   Office/Productivity
   Python Classroom
   Python Science
   Robotics
   RPM Development Tools
   Security Lab
   Sound and Video
   Sway Window Manager (supplemental packages)
   System Tools
   Text-based Internet
   Window Managers
hrom@ibm-p8-kvm-03-guest-02:~$ sudo dnf5 group list
Updating and loading repositories:
Repositories loaded.
ID                          Name                                        Installed
3d-printing                 3D Printing                                        no
admin-tools                 Administration Tools                               no
audio                       Audio Production                                   no
authoring-and-publishing    Authoring and Publishing                           no
budgie-desktop              Budgie                                             no
budgie-desktop-apps         Budgie Desktop Applications                        no
c-development               C Development Tools and Libraries                  no
cloud-infrastructure        Cloud Infrastructure                               no
cloud-management            Cloud Management Tools                             no
compiz                      Compiz                                             no
container-management        Container Management                               no
d-development               D Development Tools and Libraries                  no
design-suite                Design Suite                                       no
development-tools           Development Tools                                  no
domain-client               Domain Membership                                  no
editors                     Editors                                            no
education                   Educational Software                               no
electronic-lab              Electronic Lab                                     no
engineering-and-scientific  Engineering and Scientific                         no
freeipa-server              FreeIPA Server                                     no
games                       Games and Entertainment                            no
headless-management         Headless Management                                no
libreoffice                 LibreOffice                                        no
mate-applications           MATE Applications                                  no
milkymist                   Milkymist                                          no
network-server              Network Servers                                    no
neuron-modelling-simulators Neuron Modelling Simulators                        no
office                      Office/Productivity                                no
python-classroom            Python Classroom                                   no
python-science              Python Science                                     no
robotics-suite              Robotics                                           no
rpm-development-tools       RPM Development Tools                              no
security-lab                Security Lab                                       no
sound-and-video             Sound and Video                                    no
swaywm-extended             Sway Window Manager (supplemental packages)        no
system-tools                System Tools                                       no
text-internet               Text-based Internet                                no
window-managers             Window Managers                                    no

[j-mracek]

The problem was that DNF4 history DB data were not imported when system state directory was absent. It means that my original reproducer rm /tmp/reason2/usr/lib/sysimage/libdnf5/* did not show the problem that @jkolarik discovered (Good work Jan), but I was able to reproduce the behavior by simultaneously using DNF4 and DNF5. My patch fix only problem with the missing directory.

[jan-kolarik]

The problem was that DNF4 history DB data were not imported when system state directory was absent. It means that my original reproducer rm /tmp/reason2/usr/lib/sysimage/libdnf5/* did not show the problem that @jkolarik discovered (Good work Jan), but I was able to reproduce the behavior by simultaneously using DNF4 and DNF5. My patch fix only problem with the missing directory.

Great, thanks for looking into it and fixing the cause! :slightly_smiling_face:

[ad-on-is]

I just updatet to Fedora 41, and I have this exact issue.

dnf list --installed shows < unknown > repo for all almost all packages, while dnf4 list --installed shows the repos correctly

[evan-goode]

I just updatet to Fedora 41, and I have this exact issue.

dnf list --installed shows < unknown > repo for all almost all packages, while dnf4 list --installed shows the repos correctly

I'm curious, did you manually install dnf5 at any time before you upgraded from Fedora 40 to Fedora 41, or was the upgrade the first time DNF 5 was installed (to your knowledge)?

[ad-on-is]

@evan-goode yeah indeed, I played around with dnf5 from the unstable-repo back then on F39 (I think) ... but I used dnf4 exclusively for package management.

After upgrading to F41, I also did some cleanup, and removed the unstable repo along with some others.

[evan-goode]

Ah, yeah, that's expected then unfortunately. Luckily, users who are installing DNF 5 for the first time when upgrading to F41 won't see this bug.

[ad-on-is]

@evan-goode Ok.. and how do we, the ones who had it installed, solve the issue?

[pkly]

I just updated to F41 without installing dnf5 before the upgrade and a few of my packages list as "unknown" (eg. xorg-x11-server-xorg, xorg-x11-drv-nvidia) so this does not only affect people who did stuff with dnf5 before the upgrade.

[KalleNiemi]

I just updated to F41 without installing dnf5 before the upgrade and a few of my packages list as "unknown" (eg. xorg-x11-server-xorg, xorg-x11-drv-nvidia) so this does not only affect people who did stuff with dnf5 before the upgrade.

I have most of my packages in list as - never had dnf5 before

[dletai]

dnf reinstall \* seems to resolve this. The question is, why doesn't dist-upgrade use reinstall to bypass the issue?

[RokeJulianLockhart]

https://github.com/rpm-software-management/dnf5/issues/1379#issuecomment-2453401862

@dletai, isn't that an online action? Sounds like it could cause instability. Irrespective, I saw a recommendation to use distro-sync instead. Does anyone know which is preferrable, and why?

[dletai]

@RokeJulianLockhart distro-sync Doesn't solve the issue. If the package is already installed and same version as offered online, nothing will happen, leaving the unknown state as is.

I'm not suggesting relying on online during the upgrade, as that carries some risks, but I do suggest using it if available to get the metadata required, and simply skip if unavailable, leaving us no worse than current state.

[RokeJulianLockhart]

I'm not suggesting relying on online during the upgrade, as that carries some risks, but I do suggest using it if available to get the metadata required

@dletai, is there no way to perform the reinstallation on reboot ("offline") like dnf upgrade --offline offers?

and simply skip if unavailable, leaving us no worse than current state

Do you refer to skipping packages unavailable in the repository or skipping packages with valid origin metadata?

[dletai]

is there no way to perform the reinstallation on reboot ("offline") like dnf upgrade --offline offers? I have no idea, didn't know about --offline. Does it store the relevant metadata as well? If so, I would expect dnf system-upgrade download --offline --realeasever=... followed by dnf offline reboot (or is it dnf system-upgrade reboot --offline?) to remove the issue altogether.

and simply skip if unavailable, leaving us no worse than current state

Do you refer to skipping packages unavailable in the repository or skipping packages with valid origin metadata? I meant the first, mainly, but also if the repo is unavailable for whatever reason. I don't really care about the 2nd, as it doesn't cost me anything to get the metadata again even if it's already there.

[luksak]

I also had dnf5 installed on Fedora 40 and results in a broken database for dnf5 in Fedora 41. Sadly none of he suggested commands solve my issue. Since I'm able to restore my Fedora 40 backup: Is there a way to uninstall and remove all data of dnf5 before upgrading to Fedora 41? Or does anyone have other suggestions what I could do?

[principis]

This solved it for me:

rm -rf /usr/lib/sysimage/libdnf5
dnf4 reinstall dnf5

[luksak]

@principis Thank you! you ran that on Fedora 40? And after that you upgraded to Fedora 41? Going to try that now.

One thing worth mentioning: dnf5 dind't list any installed packages after my last try to upgrade:

$ sudo dnf5 list --installed
Updating and loading repositories:
Repositories loaded.
No matching packages to list

[principis]

@principis Thank you! you ran that on Fedora 40? And after that you upgraded to Fedora 41? Going to try that now.

I ran that on Fedora 41, I already upgraded. But removing dnf5 and /usr/lib/sysimage/libdnf5 on Fedora 40 before upgrading should suffice.

[luksak]

@principis Ok, that didn't work either. I'm still having the same situation. I tried running this before and after the upgrade. It completed without errros, but it left me in the same situation in the end.

Except for one thing: After running it on 41 for the first time, dnf5 list --installed listed my packages, but in the <unknown> repo.

Maybe I have a different issue? Should I open a new one? Is there commands I could run to provide more debug information?

[principis]

Does dnf4 list --installed show the packages? If so I would try executing the commands I mentioned previously again. There is also dnf5 check but I'm not sure what that does exactly.

[luksak]

Ok, here is the output on Fedora 40:

dnf5 check
gnome-control-center-0:46.3-1.fc40.x86_64
 missing require "gnome-control-center-filesystem = 46.3-1.fc40"
kernel-devel-matched-0:6.10.8-200.fc40.x86_64
 missing require "kernel-devel = 6.10.8-200.fc40"
libvirt-daemon-config-network-0:10.1.0-3.fc40.x86_64
 missing require "libvirt-daemon-driver-network = 10.1.0-3.fc40"
nextcloud-client-nautilus-0:3.13.2-1.fc40.x86_64
 missing require "nextcloud-client(x86-64) = 3.13.2-1.fc40"
php-0:8.3.10-1.fc40.x86_64
 missing require "php-common(x86-64) = 8.3.10-1.fc40"
php-bcmath-0:8.3.10-1.fc40.x86_64
 missing require "php-common(x86-64) = 8.3.10-1.fc40"
php-gd-0:8.3.10-1.fc40.x86_64
 missing require "php-common(x86-64) = 8.3.10-1.fc40"
python3-boto3-0:1.34.162-1.fc40.noarch
 missing require "(python3.12dist(botocore) < 1.35~~ with python3.12dist(botocore) >= 1.34.162)"
python3-devel-0:3.12.4-1.fc40.x86_64
 missing require "python3 = 3.12.4-1.fc40"
 missing require "python3-libs(x86-64) = 3.12.4-1.fc40"
vim-minimal-2:9.1.672-1.fc40.x86_64
 missing require "vim-data = 2:9.1.672-1.fc40"
Check discovered 11 problem(s) in 10 package(s)

I think don't understand the meaning of dnf5 check: It shouldn't report any problems and I should try to fix those before trying anything else?

[dletai]

I believe dnf check functionally compares the local db to the installed rpms, to verify a consistent state. All the missing requires issues (aside from the python-boto) seems to be due to an exact version requirement, and I would guess your system has a higher version installed of the so-called missing packages.

So I don't think solving these issues should have any bearing on the current issue discussed - the dnf4 vs. dnf5 metadata for already installed packages.

[luksak]

@principis Sorry, I currently don't have access to the Fedora 41 system. I'll try that once I have more time. Thank you!

@dletai Thank you for your clarification.