Closed tlaurion closed 2 months ago
Note that I have not tested upstream. I chose qusal personally, since qubes-builder was there and I needed that. Plus so many other improvements over shaker for my use cases.
But for things to evolve in the right direction, PR should be opened both ways to improve the solution for all.
The comments diff are due to my acng.conf version have been updated to Debian 12 acng.conf. Except the remove of unicode, which I did manually.
A cleaner diff can be obtained by removing comments and empty lines.
[user@dev ~/src/authored/qusal/salt/sys-cacher(main)]
% grep -vE "^(\s*(#|$))" files/server/conf/acng.conf >/tmp/q-acng.conf
[user@dev ~/src/authored/qusal/salt/sys-cacher(main)]
% grep -vE "^(\s*(#|$))" ~/src/pub/shaker/cacher/acng.conf >/tmp/s-acng.conf
[user@dev ~/src/authored/qusal/salt/sys-cacher(main)]
% diff /tmp/q-acng.conf /tmp/s-acng.conf
5,6c5,8
< Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
< Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
---
> Remap-alxrep: file:archlx_mirrors /archlinux
> Remap-debrep: https://deb.debian.org http://deb.debian.org file:deb_mirrors.gz /debian
> Remap-fedora: file:fedora_mirrors # Fedora Linux
> Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
8d9
< Remap-debrep: https://deb.debian.org http://deb.debian.org file:deb_mirrors.gz /debian
10,15d10
< Remap-fedora: file:fedora_mirrors # Fedora Linux
< Remap-fedora: file:fedora_mirrors # Fedora Linux
< Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives
< Remap-klxrep: file:kali_mirrors /kali ; file:backends_kali # Kali Linux Archives
< Remap-secdeb: security.debian.org security.debian.org/debian-security deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian-security ; deb.debian.org/debian-security security.debian.org cdn-fastly.deb.debian.org/debian-security
< Remap-sfnet: file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here
17,28c12,13
< Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
< Remap-dockerrep: https://download.docker.com http://download.docker.com
< Remap-googlerep: https://dl.google.com http://dl.google.com
< Remap-hashicorprep: https://apt.releases.hashicorp.com http://apt.releases.hashicorp.com
< Remap-kicksecuredebrep: https://deb.kicksecure.com http://deb.kicksecure.com
< Remap-launchpadrep: https://ppa.launchpad.net http://ppa.launchpad.net
< Remap-opentofurep: https://packages.opentofu.org http://packages.opentofu.org
< Remap-qubesdebrep: https://deb.qubes-os.org http://deb.qubes-os.org
< Remap-qubesyumrep: https://yum.qubes-os.org http://yum.qubes-os.org
< Remap-signalrep: https://updates.signal.org http://updates.signal.org
< Remap-syncthingrep: https://apt.syncthing.net http://apt.syncthing.net
< Remap-whonixdebrep: https://deb.whonix.org http://deb.whonix.org
---
> Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives
> Remap-secdeb: security.debian.org ; security.debian.org deb.debian.org/debian-security
32,35c17,19
< FollowIndexFileRemoval: 1
< PfilePatternEx: .*yaml.gz$|.*fedora.*arch=x86_64$|.*f[0-9]+&arch=x86_64
< VfilePatternEx: .*fedora.*updateinfo.*xml.zck$|^/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY.*|.*\?repo=fedora|.*pkg.tar.zst.sig
< DontCache: .*fedora.*updates.*updateinfo.xml.zck .*fedora.*updates.*repomd.xml
---
> PfilePatternEx: .*yaml.gz$|.*fedora.*arch=x86_64$|.*f37&arch=x86_64|.*f38&arch=x86_64|.*f39&arch=x86_64
> VfilePatternEx: .*fedora.*updateinfo.*xml.zck$|^/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY.*|.*\?repo=fedora|.*pkg.tar.zst.sig|.*archlinux.*sha256sums.txt|.*archlinux/iso.*tar.gz.sig
> DontCache: .*fedora.*updates.*updateinfo.xml.zck .*fedora.*updates.*repomd.xml
If you compare the mirrors themselves, there are some differences.
[user@dev ~/src/authored/qusal/salt/sys-cacher/files/server(main)]
% diff mirrors/fedora_mirrors ~/src/pub/shaker/cacher/fedora_mirrors
1,8d0
< # SPDX-FileCopyrightText: 2022 unman <unman@thirdeyesecurity.org>
< #
< # SPDX-License-Identifier: AGPL-3.0-or-later
< #
< # Information gathered from https://admin.fedoraproject.org/mirrormanager/
< # Link from /etc/yum.repos.d/fedora.repo:
< # https://mirrors.fedoraproject.org/metalink?repo=fedora-source-f$releasever&arch=$basearch
<
15d6
< http://dl.fedoraproject.org
17a9
> http://download-ib01.fedoraproject.org/pub/fedora/linux/
18a11
> http://fedora-archive.ip-connect.info/fedora/linux/
19a13
> http://fedora-mirror02.rbc.ru/pub/fedora/linux/
22a17
> http://fedora.ip-connect.vn.ua/linux/
28d22
< http://fedora.mirror.garr.it
30c24
< http://fedora.mirror.liteserver.nl
---
> http://fedora.mirror.liteserver.nl/
36d29
< http://fedora.mirrorservice.org/fedora/linux/
37a31
> http://forksystems.mm.fcix.net/fedora/linux/
40c34,36
< http://ftp-nyc.osuosl.org
---
> http://ftp-chi.osuosl.org/pub/fedora/linux
> http://ftp-chi.osuosl.org/pub/fedora/linux/
> http://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/
56c52
< http://ftp.linux.org.tr/fedora/updates
---
> http://ftp.linux.org.tr
61c57
< http://ftp.otenet.gr/linux/
---
> http://ftp.otenet.gr/linux/fedora/linux/
91a88
> http://mirror.dst.ca/fedora-linux/fedora/linux/
95a93
> http://mirror.fcix.net/fedora/linux/
97a96
> http://mirror.ihost.md/fedora/
101c100
< http://mirror.it4i.cz/fedora/linux/
---
> http://mirror.it4i.cz
106a106,108
> http://mirror.linux-ia64.org/fedora/fedora/linux/
> http://mirror.linux-ia64.org/fedora/linux/
> http://mirror.math.princeton.edu/pub/fedora/linux/
110a113
> http://mirror.netzwerge.de/fedora/linux/
122a126,127
> http://mirror.serverion.com/fedora/linux
> http://mirror.serverion.com/fedora/linux
123a129
> http://mirror.siena.edu/fedora/linux/
127a134
> http://mirror.stjschools.org/fedora/linux/
133a141
> http://mirror.usi.edu/pub/fedora/linux/
140a149
> http://mirror.xenyth.net/fedora/linux/
145c154,155
< http://mirrors.dotsrc.org/fedora-enchilada/linux/
---
> http://mirrors.dotsrc.org/fedora/linux
> http://mirrors.dotsrc.org/fedora/linux
147a158,159
> http://mirrors.fedoraproject.org/fedora/linux
> http://mirrors.fedoraproject.org/fedora/linux
155a168,169
> http://mirrors.rit.edu/fedora/fedora/linux
> http://mirrors.rit.edu/fedora/fedora/linux/
159c173,175
< http://mirrors.xtom.de
---
> http://mirrors.xtom.de/fedora/
> http://mirrors.xtom.ee/fedora/linux
> http://mirrors.xtom.ee/fedora/linux
160a177,180
> http://nnenix.mm.fcix.net/fedora/linux
> http://nnenix.mm.fcix.net/fedora/linux
> http://nocix.mm.fcix.net/fedora/linux/
> http://opencolo.mm.fcix.net/fedora/linux/
162d181
< http://packages.oit.ncsu.edu
165a185
> http://southfront.mm.fcix.net/fedora/linux/
166a187,189
> http://uvermont.mm.fcix.net/fedora/linux
> http://uvermont.mm.fcix.net/fedora/linux
> http://veronanetworks.mm.fcix.net/fedora/linux/
167a191
> http://volico.mm.fcix.net/fedora/linux
170a195
> http://ziply.mm.fcix.net/fedora/linux/
In any way, even updating the mirror list is no guarantee of solving the issues. Fedora mirrors are not always in sync and this is the cause of problems. Sometimes they work, sometimes doesn't.
I will see what I can do.
The zchunk errors are difficult to reproduce, I tested with Shaker's fedora_mirrors and Qusal's fedora_mirrors and both failed, I tested setting zchunk=False
on dnf.conf but that works randomly. I will update the fedora mirrors because it seems a good idea, but it does not guarantee that Fedora works well with apt-cacher-ng.
Commitment
I confirm that I have read the following resources:
Current problem (if any)
upstream: https://github.com/unman/shaker/commits/main/cacher/acng.conf downstream: https://github.com/unman/shaker/blob/main/cacher/acng.conf
Comparison of files:
Proposed solution
Keep track of upstream changes, and also do PR there so that the outcome (sys-cacher) works on QubesOS properly.
Upstream attempted to move this forward:
Discussions on sys-cacher are happening on QOS forum https://forum.qubes-os.org/t/apt-cacher-ng-and-fedora-cannot-prepare-internal-mirrorlist-status-code-403/22852 and elsewhere.
The value to a user, and who that user might be
Default template by default (next next next) on Q4.2.1 is still Fedora, and Fedora doensn't work out of the box.