libguestfs / supermin

tool for creating supermin appliances (formerly called febootstrap). PLEASE DO NOT USE GITHUB FOR ISSUES OR PULL REQUESTS. See the website for how to file a bug or contact us. http://libguestfs.org
GNU General Public License v2.0
166 stars 32 forks source link

The AUR should not be a supported platform #20

Open dvzrv opened 6 months ago

dvzrv commented 6 months ago

Hi! :wave:

I package libguestfs for Arch Linux. When investigationg https://github.com/libguestfs/libguestfs/issues/139 I noticed the following:

:: Retrieving packages...
 gcc-libs-13.2.1-5-x86_64 downloading...
 perl-5.38.2-1-x86_64 downloading...
 icu-74.2-2-x86_64 downloading...
 glibc-2.39-1-x86_64 downloading...
 systemd-255.4-2-x86_64 downloading...
 binutils-2.42-2-x86_64 downloading...
 vim-runtime-9.1.0252-1-x86_64 downloading...
 grub-2:2.12-2-x86_64 downloading...
 glib2-2.80.0-2-x86_64 downloading...
 openssl-3.2.1-1-x86_64 downloading...
 util-linux-2.40-2-x86_64 downloading...
 gnutls-3.8.5-1-x86_64 downloading...
 coreutils-9.5-1-x86_64 downloading...
 qemu-common-8.2.2-2-x86_64 downloading...
 gettext-0.22.4-1-x86_64 downloading...
 bash-5.2.026-2-x86_64 downloading...
 lvm2-2.03.23-3-x86_64 downloading...
 sqlite-3.45.2-1-x86_64 downloading...
 hwdata-0.381-1-any downloading...
 pcre2-10.43-3-x86_64 downloading...
 gawk-5.3.0-1-x86_64 downloading...
 krb5-1.21.2-2-x86_64 downloading...
 kbd-2.6.4-1-x86_64 downloading...
 linux-api-headers-6.7-1-any downloading...
 db5.3-5.3.28-4-x86_64 downloading...
 e2fsprogs-1.47.0-2-x86_64 downloading...
 ncurses-6.4_20230520-1-x86_64 downloading...
 shadow-4.15.1-2-x86_64 downloading...
 openssh-9.7p1-1-x86_64 downloading...
 systemd-libs-255.4-2-x86_64 downloading...
 curl-8.7.1-5-x86_64 downloading...
 thin-provisioning-tools-1.0.12-1-x86_64 downloading...
 cdrtools-3.02a09-5-x86_64 downloading...
 procps-ng-4.0.4-3-x86_64 downloading...
 pam-1.6.1-2-x86_64 downloading...
 wolfssl-5.7.0-1-x86_64 downloading...
 libxml2-2.12.6-1-x86_64 downloading...
 cryptsetup-2.7.2-1-x86_64 downloading...
 libcap-2.69-4-x86_64 downloading...
 libunistring-1.2-1-x86_64 downloading...
 xz-5.6.1-3-x86_64 downloading...
 libgcrypt-1.10.3-1-x86_64 downloading...
 libelf-0.191-1-x86_64 downloading...
 libarchive-3.7.3-1-x86_64 downloading...
 libp11-kit-0.25.3-1-x86_64 downloading...
 zstd-1.5.5-1-x86_64 downloading...
 util-linux-libs-2.40-2-x86_64 downloading...
 nettle-3.9.1-1-x86_64 downloading...
 gmp-6.3.0-1-x86_64 downloading...
 iptables-1:1.8.10-1-x86_64 downloading...
 mpfr-4.2.1-2-x86_64 downloading...
 findutils-4.9.0-3-x86_64 downloading...
 libnl-3.9.0-1-x86_64 downloading...
 iana-etc-20240222-1-any downloading...
error: failed retrieving file 'iana-etc-20240222-1-any.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'iptables-1:1.8.10-1-x86_64.pkg.tar.zst.sig' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'findutils-4.9.0-3-x86_64.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: too many errors from repos.archlinux.org, skipping for the remainder of this transaction
error: failed retrieving file 'libnl-3.9.0-1-x86_64.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'mpfr-4.2.1-2-x86_64.pkg.tar.zst.sig' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.
--2024-04-15 14:38:02--  https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving aur.archlinux.org (aur.archlinux.org)... 95.216.144.15, 2a01:4f9:c010:50::1
Connecting to aur.archlinux.org (aur.archlinux.org)|95.216.144.15|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-04-15 14:38:02 ERROR 404: Not Found.

--2024-04-15 14:38:02--  https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving aur.archlinux.org (aur.archlinux.org)... 95.216.144.15, 2a01:4f9:c010:50::1
Connecting to aur.archlinux.org (aur.archlinux.org)|95.216.144.15|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-04-15 14:38:02 ERROR 404: Not Found.

supermin: set -e
          umask 0000
          cd '/var/tmp/supermind89798.tmpdir/ytx0efty'
          wget 'https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz'
          tar xf 'cdrtools.tar.gz'
          cd 'cdrtools'
          /usr/bin/makepkg
          mv 'cdrtools'-*.pkg.tar.xz '/var/tmp/supermind89798.tmpdir/ytx0efty'
       : command failed, see earlier errors
make[2]: *** [Makefile:1080: stamp-supermin] Error 1
make[2]: Leaving directory '/build/libguestfs/src/libguestfs-1.52.0/appliance'
make[1]: *** [Makefile:1089: all-recursive] Error 1
make[1]: Leaving directory '/build/libguestfs/src/libguestfs-1.52.0'
make: *** [Makefile:995: all] Error 2

This appears to be implemented in https://github.com/libguestfs/supermin/blob/5a44ffc68f6eb2906e761dad63dcd207b5ca607a/src/ph_pacman.ml#L196-L234

Frankly, this is quite the horrific approach. The AUR is unsupported for a reason and should never be used in the context of distribution packages. In this particular case supermin attempts to build a package that is available in the repositories (cdrtools), but not in the AUR.

If the packages can not be retrieved, the build process should not fall back to building from unverified and untrusted sources but instead just fail! To be more specific: Please do not automatically build from the AUR at all!

rwmjones commented 6 months ago

Sure, we can drop AUR if you like. Are there other packages which are not in the base Arch distribution that we need?

dvzrv commented 6 months ago

That's a good question. Where do I need to look to find out?

Are these the relevant blocks? https://github.com/libguestfs/libguestfs/blob/6ba64125d960c820b75727e3856b0df0ed6b264c/appliance/packagelist.in#L99-L123 https://github.com/libguestfs/libguestfs/blob/6ba64125d960c820b75727e3856b0df0ed6b264c/appliance/packagelist.in#L243-L298

I know that e.g. zerofree is not packaged for the official repos, but it doesn't appear to be built from the AUR (see https://gitlab.archlinux.org/archlinux/packaging/packages/libguestfs/-/issues/1).

Could you explain which of the packages are hard dependencies?

rwmjones commented 6 months ago

Yes it's those. Supermin actually ignores packages that it doesn't know about (by design), so it's rather hard to know which ones are really required and which aren't. A number of them will just enable optional features in libguestfs but it'll work without them.

zerofree is definitely not important.

I was just wonder if there are major packages in AUR which are not in base Arch that we should know about, however if not I guess we can just try it.

dvzrv commented 6 months ago

Yes it's those.

Just to make sure: Are all those pkgs only build time dependencies? With libguestfs I always had a hard time figuring out what is build time and what is runtime dependency :S

dvzrv commented 6 months ago

The following dependencies do not exist (neither in the official repositories nor in the AUR):

The following only exist in the AUR:

rwmjones commented 6 months ago

Yes it's those.

Just to make sure: Are all those pkgs only build time dependencies? With libguestfs I always had a hard time figuring out what is build time and what is runtime dependency :S

They're both. Libguestfs builds a skeleton appliance and fills it in with files from the host filesystem at runtime (https://rwmj.wordpress.com/2014/03/08/supermin-version-5/).

Looking at the list above, I would say the only one which is important is util-linux{-ng}. I'm sure you must be packaging util-linux-ng, but probably it's been renamed as util-linux. Old util-linux was deprecated years ago. (https://en.wikipedia.org/wiki/Util-linux)