Use system dnf cache?

[mikhailnov]

Every launch of dnfdragora takes quite a lot of time to at least make cache and maybe update metadata, why can't it use system dnf cache without updating it, provided that dnf-makecache.timer is enabled by default? Just to speedup things a lot. Or maybe update metadata in the background.

[anaselli]

Are you using Group or all view? first needs more local caching in fedora since groups are not managed as in mageia.

[anaselli]

Group caching apart, "caching time" is required since those data are sent from dnf-daemon as you can see when progress bar (or reading logs) increases

[anaselli]

and that is performed in background... but dnfdragora has nothing to show yet... i wasn't able to speed up that part, ideas or help are welcome :)

[mikhailnov]

I am using Group view, all packages do have Group tag in ROSA. Yeah, I already understood, some time is required to load information about all packages. Do dnfdaemon+dnfdragora make a separate cache here? Do they use the cache that is updated by dnf-makecache.timer or a separate one?

[mikhailnov]

I installed a fresh build of dnfdragora by running sudo dnf in --refresh dnfdragota-gtk, so I had a fresh system cache at the moment of the first launch of dnfdragora, but dnfdragora started to refresh metadata on the first launch and that took a lot of time on my CPU. So, it either uses a separate cache (what for?) or incorrectly handles such case.

[anaselli]

dnfdragora has also a local "cache" with all the package and groups, though group caching should not be a problem for openmandriva if, as i think, has groups like mageia. But as said i tried to split and optimize that code without gain that much... maybe a better software engineer could help here? Blocking query where slow as well but dialgo was ugly, now dnfdragora waits for data, but time is almost the same...

[Conan-Kudo]

We're already using the system cache through dnfdaemon, so this is not actually an issue. The daemon does need to load the cache into memory and enumerate everything, which takes some time.

[mikhailnov]

4 апреля 2021 г. 17:56:06 GMT+03:00, "Neal Gompa (ニール・ゴンパ)" @.***> пишет:

We're already using the system cache through dnfdaemon, so this is not actually an issue. The daemon does need to load the cache into memory and enumerate everything, which takes some time.

Then why did it update metadata on the first launch? Metadata was about 10-30 min old

[anaselli]

Sorry i can't get what you mean.

[wojnilowicz]

Sorry i can't get what you mean.

I believe that the issue here is as follows:

STEPS TO REPRODUCE IN A TERMINAL: 1) dnf clean all (this cleans user dnf cache) 2) sudo dnf clean all (this cleans system dnf cache) 3) sudo dnf makecache (this refreshes system dnf cache by downloading full metadata) 4) dnfdragora (this indirectly refreshes user dnf cache by downloading full metadata again)

BEHAVIOR: dnfdragora refreshes user dnf cache by downloading full metadata again which takes significant amount of time.

EXPECTED BEHAVIOR: dnfdragora should use system dnf cache or allow choosing between using either user or system dnf cache.

ADDITIONAL INFORMATION: 1) Fedora 35 KDE 2) sudo systemctl disable dnf-makecache.timer 3) sudo systemctl disable dnf-system-upgrade 4) sudo systemctl stop dnfdaemon 5) sudo dnf remove dnfdragora-updater

Please consider reopening, and fixing this issue.

[anaselli]

As far as I can say with commit ce29aca45d14f9f45d512d3fc5eb6b35db1ab19d and 674f768e10d7f5d90a9069d2a9c89da7a8239e09 We improved metadata management, maybe bugs can be present. We need help to understand how to change the behavior if it is wrong. Patches are also welcome

[wojnilowicz]

As far as I understand, the commits that you mention, don't touch the issue explained here. Maybe you could require a password for refreshing metadata similarly to the way you require a password for installing/updating/removing a package.

[anaselli]

password is needed only if dnfdaemon needs right escalation. Check should be here and called here.

[wojnilowicz]

Password (or root permissions) is also needed if you want to refresh dnf system cache. You could make a checkbox in the settings of dnfdragora to always do refreshing with root permissions. This way you would make operating of dnfdragora less irritating for some of us.

[anaselli]

No it isn't needed as far as i know. As said password is requested by right escalation in dnfdaemon, so if dnfdaemon API does not require it, dnfdragora does not ask for password too. Metadata is updated every 48 hours (see system settings and related metadata section into dnfdragora.yaml config file). Default value is 48 hours (update_interval), so every two days since "last_update" entry, the check is performed.

Above mentioned commits should bypass this by using dnf-makecache.timer that updates metadata and dnfdragora should assume all is always updated.

As said a bug could be present but it should work like that

[wojnilowicz]

Ok, so you have a bug with steps to reproduce it, and at least two independent persons who report faulty behavior. You cannot reproduce the faulty behavior yourself and everything works correctly on your end, true?

BTW. dnf-makecache.timer is disabled on my system.

[anaselli]

Honestly i can't see that. Or maybe my system is not so slow, and i have dnf-makecache.timer enabled.

So i can't reproduce what you say, and i don't have a test case to get the issue.

Maybe you can try to comment the line, I'm not sure it is correct, but try it and let me know, i will check if there is a reason to get it there or some where else...

[wojnilowicz]

Ok. No difference after commenting the line you mentioned. What I think helped is setting "Metadata expire time (hours)" to zero. Then after:

1. dnf clean all
2. sudo dnf clean all
3. sudo dnf makecache dnfdragora doesn't redownload the metadata again, and is still able to lists packages.

Regarding you are not able to reproduce the issue: 1) maybe for the sake of testing you could disable dnf-makecache.timer and enable it afterwards 2) if your system is not so slow then maybe you could measure data downloaded. At clean caches, 'sudo dnf makecache' downloads over 100 MB. If you're not hitting this bug then dnfdragora should download 0 MB. In my case it downloads also over 100 MB.

BTW. Shouldn't settings->metadata->last_update in ~/.config/dnfdragora.yaml be always updated when metadata are redownloaded? Mine is like from 2022-03-26, although I did the tests with metadata recently.

[anaselli]

hmm maybe the bug is there then. Have you refreshed metadata from dnfdragora by any chance? the date should change to that time and if you lunch dnfdragora again should not reload metadata, please confirm it.

[wojnilowicz]

Refreshing metadata from dnfdragora causes:

1. over 100 MB data to be downloaded
2. no update in ~/.config/dnfdragora.yaml

BTW. dnfdragora doesn't redownload metadata at its every launch. Once it refreshed its metadata, it doesn't need to refresh it again if I start it e.g. 5 minutes later.

[anaselli]

1. over 100 MB data to be downloaded

Ok it doesn't add any additional information

2. no update in ~/.config/dnfdragora.yaml That is odd here it works. Which dnfdragora version are you using?

BTW. dnfdragora doesn't redownload metadata at its every launch. Once it refreshed its metadata, it doesn't need to refresh it again if I start it e.g. 5 minutes later. that is odd too, i'd expect hours you set (48 by default, need to check what 0 does really though).

Said that, i'd like to know how dnfdragora can know when a refresh has been performed outside by hands, since you don't use dnf-makecache.timer and no one can set the date but dnfdragora itself (need to check if that realy work though).

[wojnilowicz]

1. over 100 MB data to be downloaded

Ok it doesn't add any additional information

I don't know if you know but after first 'sudo dnf makecache' (over 100 MB downloaded) subsequent 'sudo dnf makecache' downloads only a couple of kilobytes (unless you pass "--refresh" to it). I would expect dnfdragora to perform similarly, instead of redownloading whole 100 MB again.

2. no update in ~/.config/dnfdragora.yaml

That is odd here it works. Which dnfdragora version are you using?

2.1.0

BTW. dnfdragora doesn't redownload metadata at its every launch. Once it refreshed its metadata, it doesn't need to refresh it again if I start it e.g. 5 minutes later. that is odd too, i'd expect hours you set (48 by default, need to check what 0 does really though).

Said that, i'd like to know how dnfdragora can know when a refresh has been performed outside by hands, since you don't use dnf-makecache.timer and no one can set the date but dnfdragora itself (need to check if that realy work though).

True, I don't use dnf-makecache.timer but I don't rely on 'sudo dnf makecache' exclusively. My use case is:

1. sudo dnf update (when I want to update packages because it's the fastest for me)
2. dnfdragora (when I want to install an unknown package because it's the most convenient for me) The thing is that those two approaches doesn't share a common cache.

[Conan-Kudo]

dnfdaemon does not require privilege escalation to update the system cache (it already runs as root). It already uses the regular dnf cache.

[anaselli]

@Conan-Kudo how can be shared metadata update date between dnf and dnfdragora? I think that is not possible, am i wrong? We could allow to disable metadata expiration check, not as default behavior though.

[anaselli]

@wojnilowicz i can't remember from which version is in, but i added a check time ago that skips meta data expiration evaluation if _updateinterval: is <= 0, so if you set it as 0 into option dialog or by hands into configuration file should do what you wanted. It isn't documented because it was a check to avoid mistakes in counting but well it seems what you wanted, let us know if it works. I need to check expiration evaluation though, i don't think it works as expected... but that is another issue :)

[anaselli]

I added some log information to understand if for dnfdragora point of view metadata should be downloaded or not

[Conan-Kudo]

@Conan-Kudo how can be shared metadata update date between dnf and dnfdragora? I think that is not possible, am i wrong? We could allow to disable metadata expiration check, not as default behavior though.

The daemon always runs privileged as root, so it has access to the DNF cache.

[anaselli]

The daemon always runs privileged as root, so it has access to the DNF cache.

@Conan-Kudo I meant if there APIs already :D But yes we shouid move this discussion there and see how can we get and make available data...

[wojnilowicz]

@wojnilowicz i can't remember from which version is in, but i added a check time ago that skips meta data expiration evaluation if _updateinterval: is <= 0, so if you set it as 0 into option dialog or by hands into configuration file should do what you wanted. It isn't documented because it was a check to avoid mistakes in counting but well it seems what you wanted, let us know if it works. I need to check expiration evaluation though, i don't think it works as expected... but that is another issue :)

I don't know. I started dnfdragora today and it seems to work if _updateinterval: is <= 0 because it doesn't download over 100 MB of the data, but in the status bar it still displays that it tries to download the repo data. Maybe it's just for visual purposes.

[anaselli]

hmm are you sure that what you see is the package caching instead? that is performed to show groups and other things.

[anaselli]

@Conan-Kudo i don't know that much dnfdaemon code, but what a way could be when dnfdaemon starts:

1. checking if dnf make cache is active (no action if it is)
2. otherwise checking dnf metadata in the same manner dnf does and if considerd expired (should be done in the same manner as dnf too) run makecache, thiis should send info to dnfdragora as it were invoked by it.

That way should not need to add any new API, how does that sound?

[Conan-Kudo]

Seems sensible to me.

[anaselli]

ok reopening this issue, let's hope to find time soon to work on it

[wojnilowicz]

hmm are you sure that what you see is the package caching instead? that is performed to show groups and other things.

I don't know. It says "Fedora 35 - x86_64 - Updates" above the progress bar, whatever that means. Only around 7 MB has been downloaded.

[anaselli]

that is it, so. Unfortunately dnfdragora does not ask for a refresh, i wonder if it is required by some other API usage or called implicitally for some reason...