anatol
commented
3 years ago cc @Focshole
Closed HidingCherry closed 2 years ago
anatol
commented
3 years ago cc @Focshole
Focshole
commented
3 years ago I'm not sure that this is a bug, so I'll start by explain how prefetching is supposed to work. The prefetching mechanism is thought to prefetch packages which have been requested before. In your case, those packages are:
Now, in a few days i do not expect any packages to be updated in such a situation (because it is kinda rare that the same package gets updated twice in a small period of time, maybe it could happen with testing repos, i don't know).
Anyway, if you want to know which packages have been configured for prefetching, you can do so by inspecting the sqlite-pkg-cache.db sqlite3 db file, you should be able to know which are the packages which should be prefetched. To do so, there are gui tools to inspect sqlite3 dbs, you could copy the db file wherever you want and inspect it freely.
Now, there could still be a bug in there. At the moment I have tested it with a single repos entry and it works for me, I have never seen a package not being updated. If this happens for you, please provide me some extra info (e.g. the package affected which had been requested but not prefetched on an update), so that I could debug/replicate the issue more in depth.
Now, If you want that ALL the packages you have installed on your pc/laptop/whatever would be prefetched, I had thought about 2 approaches:
Slow but easy approach, do just once something like:
pacman -Scc
for pkg in $(pacman -Q | cut -d' ' -f1); do
pacman -Sw --noconfirm $pkg
doneThat is, remove all cached packages, then ask pacoloco to serve them all ( -Sw should simply download them), so that in this way its db has the whole list. It should be a very long process, so if you plan to do so, take that into account.
I had thought about having an initialization script for this, but it would leave the db in a incoherent state (it thinks that a package is there but it actually isn't) and also the initialization script may lead to many more incoherent states in the db (if a bug would be in the initialization script, it could be very hard to sort it out). It won't crash, but it would be in a "dirty" state.
As a better approach, you could move all of your cached packages into the proper pacoloco directories, then launch the same script written before (i haven't tried it so far, it is kinda hacky but it should work), so that you could use your local network speed to redownload those packages
HidingCherry
commented
3 years ago Well, then my sqlite db makes no sense...
ls /[…]/@pacoloco/pkgs/arch_x86-64/ | wc -l
496(including .sig files, so basically 248 packages)
But the sqlite file shows me 59 entries in the packages table. Maybe I should delete the db and look if a fresh one helps?
HidingCherry
commented
3 years ago Example:
ls -l /[…]/@pacoloco/pkgs/arch_x86-64/dkms-3.0.1-1-any.pkg.tar.zst*
-rw-r--r-- 1 pacoloco pacoloco 36876 Nov 15 12:46 /[…]/@pacoloco/pkgs/arch_x86-64/dkms-3.0.1-1-any.pkg.tar.zst
-rw-r--r-- 1 pacoloco pacoloco 558 Nov 15 12:46 /[…]/@pacoloco/pkgs/arch_x86-64/dkms-3.0.1-1-any.pkg.tar.zst.sigThere is a new update for that (version 3.0.2-1). But no entry for dkms inside the db.
Focshole
commented
3 years ago If you remember, you had the version without prefetching, which means that there was no db, so none of the packages requested before that update were actually recorded into the db. So, the packages included in the db should be the ones requested after the update EDIT: sorry I had been imprecise on the "pacoloco up and running"
Focshole
commented
3 years ago Well, then my sqlite db makes no sense...
ls /[…]/@pacoloco/pkgs/arch_x86-64/ | wc -l 496(including .sig files, so basically 248 packages)
But the sqlite file shows me 59 entries in the packages table. Maybe I should delete the db and look if a fresh one helps?
It shouldn't help, the db didn't track the packages which had been requested before the update you did
HidingCherry
commented
3 years ago Yeah, I see - makes sense now.
For your script - this one might be better: echo $(pacman -Q | cut -d' ' -f1)| xargs paru -Sw --noconfirm
But there is an issue if you are using packages which have been removed from aur or are custom build.
EDIT:
For removed packages use sed 's/PACKAGE//' e.g.:
echo $(pacman -Q | cut -d' ' -f1) | sed 's/PACKAGE//' | xargs paru -Sw --noconfirm
you can put several sed in between.
HidingCherry
commented
3 years ago I'll download each package once and see next time (in a few days) if they have been prefetched. If it works, I'll close this issue.
Focshole
commented
3 years ago Yeah, I see - makes sense now.
For your script - this one might be better:
echo $(pacman -Q | cut -d' ' -f1)| xargs paru -Sw --noconfirmBut there is an issue if you are using packages which have been removed from aur or are custom build.EDIT: For removed packages use
sed 's/PACKAGE//'e.g.:echo $(pacman -Q | cut -d' ' -f1) | sed 's/PACKAGE//' | xargs paru -Sw --noconfirmyou can put several sed in between.
Yeah, I didn't consider custom built packages very much for prefetching, due to the fact that if there is no your_custom_repo.db file, I cannot know if they had been updated. It would act like a caching server with those at most.
If there is a db and you have an issue with those, you would most likely find some does not match regex alike error message.
I'll download each package once and see next time (in a few days) if they have been prefetched. If it works, I'll close this issue.
Yes please, keep us updated. There should be no need to delete the sqlite db (otherwise that would be another issue).
HidingCherry
commented
3 years ago It works - but I noticed a weird behavior. It prefetches some packages over and over again.
It looks like only the aarch64 architecture is problematic. e.g.:
Nov 22 01:33:33: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 01:33:34: prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 01:33:34: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:34: prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 22 01:33:37: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar
Nov 22 01:33:37: prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 09:31:10: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 09:31:10: prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 09:31:11: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 09:31:11: prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 22 16:32:37: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 16:32:37: prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 16:32:38: pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:38: prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signatureAs you see, pacoloco downloaded all the possible extensions of each package (.pkg.tar.xz, .pkg.tar, .pkg.tar.xz) because it doesn't know the required extension it has to download from upstream repositories (or to be precise, it doesn't store that info when a package gets downloaded). I don't think that it is an optimal approach but I think that it should be fine (it could be improved, that's true).
Now, I have no clue on why does it prefetch every time that package. That's weird. Does it happen for other packages too? Does pacoloco own/can edit the sqlite-pkg-cache.db file ? What about the arch_aarch64 directory?
HidingCherry
commented
3 years ago # ls -l /[…]/@pacoloco/
total 4468
drwxr-xr-x 1 pacoloco pacoloco 86 Nov 7 17:55 pkgs
-rw-r--r-- 1 pacoloco pacoloco 4575232 Nov 23 16:32 sqlite-pkg-cache.db
drwxr-xr-x 1 pacoloco pacoloco 28 Nov 23 16:30 tmp-db
# ls -l /[…]/@pacoloco/pkgs/
total 0
drwxr-xr-x 1 pacoloco pacoloco 69854 Nov 23 16:32 arch_aarch64
drwxr-xr-x 1 pacoloco pacoloco 254528 Nov 23 16:32 arch_x86-64
drwxr-xr-x 1 pacoloco pacoloco 2080 Nov 21 21:02 blackarch
drwxr-xr-x 1 pacoloco pacoloco 3448 Nov 21 21:02 endeavourosThere are way more packages which are downloaded repeatedly - only aarch64 packages.
I've copied the relevant logs - do you need the database? Something else I noticed - sometimes the .tar package gets downloaded - sometimes not.
Nov 22 01:33:03 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 22 01:33:05 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 22 01:33:06 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:06 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 22 01:33:10 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 22 01:33:11 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 22 01:33:11 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:11 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 22 01:33:13 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar
Nov 22 01:33:13 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 22 01:33:15 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 22 01:33:15 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 22 01:33:16 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:16 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 22 01:33:20 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 22 01:33:20 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 22 01:33:21 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:21 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 22 01:33:24 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 22 01:33:24 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 22 01:33:25 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:25 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 22 01:33:29 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 22 01:33:29 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 22 01:33:30 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:30 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 22 01:33:32 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar
Nov 22 01:33:32 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 22 01:33:33 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 01:33:34 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 01:33:34 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 01:33:34 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 22 01:33:37 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar
Nov 22 01:33:37 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 01:33:37 prefetch.go:275: Finished prefetching packages!
Nov 22 01:33:37 prefetch.go:276: Finished prefetching routine!
Nov 22 01:33:37 prefetch.go:45: On 2021-11-22 09:29:59.999999417 +0100 CET m=+123788.400574290 the prefetching routine will be run again
Nov 22 09:30:40 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 22 09:30:42 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 22 09:30:43 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 09:30:43 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 22 09:30:47 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 22 09:30:47 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 22 09:30:48 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 09:30:48 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 22 09:30:51 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 22 09:30:51 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 22 09:30:52 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 22 09:30:52 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 22 09:30:56 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 22 09:30:56 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 22 09:30:57 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 22 09:30:57 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 22 09:31:01 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 22 09:31:01 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 22 09:31:02 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 22 09:31:02 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 22 09:31:05 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 22 09:31:05 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 22 09:31:06 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 22 09:31:06 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 22 09:31:10 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 09:31:10 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 09:31:11 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 09:31:11 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 22 09:31:14 prefetch.go:275: Finished prefetching packages!
Nov 22 09:31:14 prefetch.go:276: Finished prefetching routine!
Nov 22 09:31:14 prefetch.go:45: On 2021-11-22 16:29:59.999999417 +0100 CET m=+148988.400574290 the prefetching routine will be run again
Nov 22 16:32:06 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 22 16:32:08 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 22 16:32:09 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:09 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 22 16:32:13 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 22 16:32:13 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 22 16:32:14 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:14 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 22 16:32:18 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 22 16:32:18 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 22 16:32:19 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:19 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 22 16:32:23 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 22 16:32:23 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 22 16:32:24 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:24 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 22 16:32:26 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar
Nov 22 16:32:26 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 22 16:32:27 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 22 16:32:27 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 22 16:32:28 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:28 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 22 16:32:30 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar
Nov 22 16:32:30 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 22 16:32:32 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 22 16:32:32 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 22 16:32:33 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:33 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 22 16:32:37 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 22 16:32:37 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 22 16:32:38 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 22 16:32:38 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 22 16:32:40 prefetch.go:275: Finished prefetching packages!
Nov 22 16:32:40 prefetch.go:276: Finished prefetching routine!
Nov 22 16:32:40 prefetch.go:45: On 2021-11-23 01:29:59.999999417 +0100 CET m=+181388.400574290 the prefetching routine will be run again
Nov 23 01:31:03 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 23 01:31:05 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 23 01:31:06 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:06 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 23 01:31:08 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar
Nov 23 01:31:08 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 23 01:31:09 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 23 01:31:09 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 01:31:10 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:10 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 23 01:31:12 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar
Nov 23 01:31:12 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 01:31:14 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 23 01:31:14 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 23 01:31:15 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:15 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 23 01:31:19 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 23 01:31:19 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 23 01:31:20 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:20 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 23 01:31:23 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 23 01:31:23 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 23 01:31:24 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:24 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 23 01:31:28 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 23 01:31:28 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 01:31:29 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:29 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 23 01:31:31 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar
Nov 23 01:31:31 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 01:31:32 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 23 01:31:32 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 23 01:31:33 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 01:31:33 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 23 01:31:35 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar
Nov 23 01:31:35 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 23 01:31:36 prefetch.go:275: Finished prefetching packages!
Nov 23 01:31:36 prefetch.go:276: Finished prefetching routine!
Nov 23 01:31:36 prefetch.go:45: On 2021-11-23 09:29:59.999999417 +0100 CET m=+210188.400574289 the prefetching routine will be run again
Nov 23 09:30:59 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 23 09:31:01 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 23 09:31:02 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:02 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 23 09:31:06 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 23 09:31:06 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 09:31:07 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:07 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 23 09:31:09 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar
Nov 23 09:31:09 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 09:31:11 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 23 09:31:11 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 23 09:31:12 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:12 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 23 09:31:15 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 23 09:31:15 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 23 09:31:16 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:16 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 23 09:31:20 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 23 09:31:20 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 23 09:31:21 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:21 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 23 09:31:25 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 23 09:31:25 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 09:31:26 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:26 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 23 09:31:28 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar
Nov 23 09:31:28 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 09:31:30 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 23 09:31:30 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 23 09:31:30 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 09:31:30 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 23 09:31:32 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar
Nov 23 09:31:32 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 23 09:31:33 prefetch.go:275: Finished prefetching packages!
Nov 23 09:31:33 prefetch.go:276: Finished prefetching routine!
Nov 23 09:31:33 prefetch.go:45: On 2021-11-23 16:29:59.999999709 +0100 CET m=+235388.400574290 the prefetching routine will be run again
Nov 23 16:32:14 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz
Nov 23 16:32:16 prefetch.go:252: Successfully prefetched systemd-aarch64 package
Nov 23 16:32:17 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:17 prefetch.go:250: Successfully prefetched systemd-aarch64 signature
Nov 23 16:32:21 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz
Nov 23 16:32:21 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 16:32:22 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:22 prefetch.go:250: Successfully prefetched systemd-libs-aarch64 signature
Nov 23 16:32:24 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-libs-249.7-1-aarch64.pkg.tar
Nov 23 16:32:24 prefetch.go:252: Successfully prefetched systemd-libs-aarch64 package
Nov 23 16:32:26 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz
Nov 23 16:32:26 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 23 16:32:27 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:27 prefetch.go:250: Successfully prefetched nftables-aarch64 signature
Nov 23 16:32:29 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.1-1-aarch64.pkg.tar
Nov 23 16:32:29 prefetch.go:252: Successfully prefetched nftables-aarch64 package
Nov 23 16:32:30 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz
Nov 23 16:32:30 prefetch.go:252: Successfully prefetched crun-aarch64 package
Nov 23 16:32:31 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/community/crun-1.3-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:31 prefetch.go:250: Successfully prefetched crun-aarch64 signature
Nov 23 16:32:35 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz
Nov 23 16:32:35 prefetch.go:252: Successfully prefetched libcap-aarch64 package
Nov 23 16:32:36 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libcap-2.61-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:36 prefetch.go:250: Successfully prefetched libcap-aarch64 signature
Nov 23 16:32:40 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz
Nov 23 16:32:40 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 16:32:41 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:41 prefetch.go:250: Successfully prefetched libnftnl-aarch64 signature
Nov 23 16:32:43 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.1-1-aarch64.pkg.tar
Nov 23 16:32:43 prefetch.go:252: Successfully prefetched libnftnl-aarch64 package
Nov 23 16:32:44 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
Nov 23 16:32:44 prefetch.go:252: Successfully prefetched systemd-sysvcompat-aarch64 package
Nov 23 16:32:45 pacoloco.go:362: downloading http://mirror.archlinuxarm.org/aarch64/core/systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig
Nov 23 16:32:45 prefetch.go:250: Successfully prefetched systemd-sysvcompat-aarch64 signature
Nov 23 16:32:48 prefetch.go:275: Finished prefetching packages!
Nov 23 16:32:48 prefetch.go:276: Finished prefetching routine!
Nov 23 16:32:48 prefetch.go:45: On 2021-11-24 01:29:59.999999708 +0100 CET m=+267788.400574289 the prefetching routine will be run againI think the problems are old packages from before prefetch has been activated:
# ls systemd-*
systemd-249.5-3-aarch64.pkg.tar.xz
systemd-249.5-3-aarch64.pkg.tar.xz.sig
systemd-249.7-1-aarch64.pkg.tar
systemd-249.7-1-aarch64.pkg.tar.xz
systemd-249.7-1-aarch64.pkg.tar.xz.sig
systemd-libs-249.5-3-aarch64.pkg.tar.xz
systemd-libs-249.5-3-aarch64.pkg.tar.xz.sig
systemd-libs-249.7-1-aarch64.pkg.tar
systemd-libs-249.7-1-aarch64.pkg.tar.xz
systemd-libs-249.7-1-aarch64.pkg.tar.xz.sig
systemd-swap-4.4.0-2-any.pkg.tar.xz
systemd-swap-4.4.0-2-any.pkg.tar.xz.sig
systemd-sysvcompat-249.5-3-aarch64.pkg.tar.xz
systemd-sysvcompat-249.5-3-aarch64.pkg.tar.xz.sig
systemd-sysvcompat-249.7-1-aarch64.pkg.tar
systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz
systemd-sysvcompat-249.7-1-aarch64.pkg.tar.xz.sig# ls -l systemd-249.*
-rw-r--r-- 1 pacoloco pacoloco 6309756 Oct 29 18:13 systemd-249.5-3-aarch64.pkg.tar.xz
-rw-r--r-- 1 pacoloco pacoloco 566 Oct 29 18:13 systemd-249.5-3-aarch64.pkg.tar.xz.sig
-rw-r--r-- 1 pacoloco pacoloco 566 Nov 22 00:39 systemd-249.7-1-aarch64.pkg.tar
-rw-r--r-- 1 pacoloco pacoloco 6309668 Nov 22 00:38 systemd-249.7-1-aarch64.pkg.tar.xz
-rw-r--r-- 1 pacoloco pacoloco 566 Nov 22 00:39 systemd-249.7-1-aarch64.pkg.tar.xz.sigEDIT: I was wrong, there were just a few older systemd packages (the mentioned above) - the others didn't have more versions...
Focshole
commented
3 years ago I'd have to check how does pacman know which extension should it try to download first, so that I can mimic the same behaviour on pacoloco. So far the logic is extremely simple, to avoid issues pacoloco tries to download all possible extensions. Probably it should stop when it finds the first ".whatever" package extension that pacman would try to download and eventually the relative ".whatever.sig" signature it finds.
HidingCherry
commented
3 years ago You should better save, which extension is being used per repo (maybe even several extensions) - while my notebook (x86_64) uses zst, my aarch64 system uses obviously xz. I think the latter one just gets what is being provided by the server, if not zst.
Focshole
commented
2 years ago You should better save, which extension is being used per repo (maybe even several extensions) - while my notebook (x86_64) uses zst, my aarch64 system uses obviously xz. I think the latter one just gets what is being provided by the server, if not zst.
I thought about it. I do not like this kind of solution because it does not scale well. Let me explain:
Mimicking what pacman would do should be the proper approach , knowing that pacoloco relies on pacman (also, if you have custom packages that you self build, you should add them as an extra repo in pacman configuration imho ). It would be simple and stupid, which should work.
Edit: It seems that the extension is in the db file which i do parse. I'll update the code to take the extension into account.
HidingCherry
commented
2 years ago
- If a mirror serves only .tar, pacoloco would prefetch only tars. If then the mirror starts to offer a more compressed format (let's say, .xz as an example), pacman would require a .xz package but pacoloco wouldn't find it, so it does a cache miss.
Wouldn't that mean, that pacoloco would begin to cache those too, since now .xz has been added to the suffix-list for that repo and it would cache both (tar, xz)?
- Also, this approach would require to keep track of a lot more stuff and change quite a lot of code (not gonna lie, this is the main issue for me :D )
Yeah, I understand that - it's nothing important though - my notebook gets its updates pretty fast and I am happy so far.
Mimicking what pacman would do should be the proper approach , knowing that pacoloco relies on pacman (also, if you have custom packages that you self build, you should add them as an extra repo in pacman configuration imho ). It would be simple and stupid, which should work.
Well - I don't care about custom packages (some of my packages are either from aur or have been deleted from aur or the official repos - I have to clean up). Those should be correctly served by the admin in the first place. Then pacoloco should also be able to handle that.
Edit: It seems that the extension is in the db file which i do parse. I'll update the code to take the extension into account.
Ah, nice - I tried to find the code in pacman (I can read/write C, but I am not really experienced) - no luck with that :/
Focshole
commented
2 years ago Ah, nice - I tried to find the code in pacman (I can read/write C, but I am not really experienced) - no luck with that :/
There were no references in pacman source code, there were refs in their imported libraries (I didn't check them honestly). Then I randomly inspected a core.db file and I realized that there was the extension shown in desc.
Anyway, this is still something I should fix because: 1) It uses much more persistent memory if a repository contains mixed versions 2) It downloads more packages than it should, which is a considerable issue (taking into account that this project is thought for people with poor internet connection)
At least it doesn't happen for most mirrors.
Right now I am very busy, it should be a short fix but still I cannot guarantee any ETA for fixing this.
I think pacoloco only prefetches the databases, but never any package. I use an aarch64 (armv8) system, but I doubt that this is the issue.
This system was down for several days, so it couldn't have downloaded/updated any packages for this time - thus there must've been some updates. After updating today I noticed that some dbs were cached, but none of the packages.
pacoloco.yaml:
journal: