Several packages still using .xz compressed source code tarballs

[ikazuhiro]

Describe the bug

Sine https://github.com/openwrt/openwrt/commit/bab3ae2ee7656600a185f4cef11cef94389023af, OpenWrt prefers .gz tarball than .xz for tools. But packages in package directory and package repository still seem to prefer .xz. As a result, dl directory contains many tarballs which have the same content. For example, I can see following redundant tarballs in my dl directory.

autoconf-2.71.tar.gz
autoconf-2.71.tar.xz
automake-1.16.5.tar.gz
automake-1.16.5.tar.xz
e2fsprogs-1.47.0.tar.gz
e2fsprogs-1.47.0.tar.xz
expat-2.6.2.tar.gz
expat-2.6.2.tar.xz
libtool-2.4.7.tar.gz
libtool-2.4.7.tar.xz
patch-2.7.6.tar.gz
patch-2.7.6.tar.xz
zlib-1.3.1.tar.gz
zlib-1.3.1.tar.xz

OpenWrt version

master

OpenWrt release

master

OpenWrt target/subtarget

all

Device

host

Image kind

Self-built image

Steps to reproduce

No response

Actual behaviour

No response

Expected behaviour

No response

Additional info

No response

Diffconfig

No response

Terms

• [X] I am reporting an issue for OpenWrt, not an unsupported fork.

[github-actions[bot]]

Invalid Version reported. `` Is this from a clean repository?

[github-actions[bot]]

Invalid Release reported. `` Is this from a clean repository?

[github-actions[bot]]

Invalid Target/Subtarget reported. `` Is this from a supported device?

[ynezz]

This is not a bug, avoiding xz is intentional after the recent xz utils drama, see bab3ae2ee7656 and 706f0e395 as examples.

[ikazuhiro]

Thank you for your replay. Yes, I know xz related issues, and my question is whether packages remain prefering .xz tarballs. I think there is no advantage to download .xz tarballs which we have already downloaded as other format.

[ynezz]

As you can see in the linked https://github.com/openwrt/openwrt/commit/bab3ae2ee7656600a185f4cef11cef94389023af the .gz or .bz2 archives are now preferred, thus if something still downloads autoconf-2.71.tar.xz then its a bug and if you want that fixed, you need to provide more details, like for example reproducer or complete log.

[ikazuhiro]

As I wrote, many packages in package directory and packages repository seems to use .xz tarballs.

$ find package/ -type f -exec grep -B 5 -E '^PKG_SOURCE:=.*\.xz$' {} + | grep -E 'PKG_(NAME|SOURCE):='

(snip)
package/utils/e2fsprogs/Makefile-PKG_NAME:=e2fsprogs
package/utils/e2fsprogs/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
package/utils/util-linux/Makefile-PKG_NAME:=util-linux
package/utils/util-linux/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
package/libs/libtool/Makefile-PKG_NAME:=libtool
package/libs/libtool/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
package/libs/mpfr/Makefile-PKG_NAME:=mpfr
package/libs/mpfr/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
package/libs/zlib/Makefile-PKG_NAME:=zlib
package/libs/zlib/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
package/libs/gmp/Makefile-PKG_NAME:=gmp
package/libs/gmp/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION)$(PKG_REVISION).tar.xz

In packages repository,

$ find . -type f -exec grep -B 5 -E '^PKG_SOURCE:=.*\.xz$' {} + | grep -E 'PKG_(NAME|SOURCE):='

(snip)
./libs/expat/Makefile-PKG_NAME:=expat
./libs/expat/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/m4/Makefile-PKG_NAME:=m4
./devel/m4/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/pkgconf/Makefile-PKG_NAME:=pkgconf
./devel/pkgconf/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/automake/Makefile-PKG_NAME:=automake
./devel/automake/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/patch/Makefile-PKG_NAME:=patch
./devel/patch/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/bison/Makefile-PKG_NAME:=bison
./devel/bison/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/libtool-bin/Makefile-PKG_NAME:=libtool
./devel/libtool-bin/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./devel/autoconf/Makefile-PKG_NAME:=autoconf
./devel/autoconf/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./utils/sed/Makefile-PKG_NAME:=sed
./utils/sed/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./utils/tar/Makefile-PKG_NAME:=tar
./utils/tar/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz
./utils/coreutils/Makefile-PKG_NAME:=coreutils
./utils/coreutils/Makefile:PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz

You can see .xz tarballs are downloaded when you build above pacakges. If I changed PKG_SOURCE and PKG_HASH values to the same with tools/.../Makefile, tarballs for tools were shared.

[BKPepe]

As you can see in the linked openwrt/openwrt@bab3ae2 the .gz or .bz2 archives are now preferred, thus if something still downloads autoconf-2.71.tar.xz then its a bug and if you want that fixed, you need to provide more details, like for example reproducer or complete log.

Is this your opinion, or is this the official statement from OpenWrt core members? I've asked on the mailing list ^1 about this, but no one responded to it except @robimarko. For me, it makes sense to get away from .tar.xz, but other package maintainers are not united about this and as you can see, recently in PR https://github.com/openwrt/packages/pull/24299, it was changed from .tar.gz to .tar.xz for several packages without proper reason, though except that .tar.xz is preferred and mentioned in one file. 🤷 . Next good example is https://github.com/openwrt/packages/pull/24218

[diizzyy]

I don't really see the issue with tar.xz tarballs especially since there have been lots of eyes on xz-utils since. Just grab whatever is the smallest tarball and go with it, it's not something other distros have enforced either to my knowledge.

[BKPepe]

No conclusion was made in the referenced PR (iputils), where it was also discussed. Closing.