Preparations to switch to APK package manager

aparcar commented 8 months ago

Hi all, some fellow developers and me worked for some time now on making APK the new package manager for OpenWrt, replacing the unmaintained OPKG fork we've been using for the longest time.

APK is actively developed and used in multiple other distributions, e.g. Alpine Linux 🎉

While there is till some work ahead, I'd like to prepare everyone who maintains a package to verify that the PKG_VERSION follows ~~Semantic Versioning~~ <major>.<minor>.<fixup>[.<sub1>...]. APK uses a deterministic algorithm to compare versions and does not like random strings, except a valid hash prefixed with a ~.

If in doubt, please use the Docker container below to verify the used version is valid:

docker run --rm -it ghcr.io/aparcar/apk-valid-version <VERSION> [<VERSION> ...]

It will print whatever version is not valid, if you get a zero exit code, you're fine.

Please feel free to reach out for assistance and have a look at the core migration of versions.

aparcar commented 8 months ago

Below packages that would cause trouble. Please bear in mind that PKG_RELEASE is soonish prefixed with an r and may only contain a number.

Naughty list:

[x] bmx6 0.1-alpha-r0 (package dropped)
[x] bmx6-json 0.1-alpha-r0 (package dropped)
[x] bmx6-sms 0.1-alpha-r0 (package dropped)
[x] bmx6-table 0.1-alpha-r0 (package dropped)
[x] bmx6-uci-config 0.1-alpha-r0 (package dropped)
[ ] cjdns v21.1-r4
[ ] cjdns-tests v21.1-r4
[x] ~~kmod-batman-adv <LINUX_VERSION>.2023.0-r0~~ 6.1.82.2024.1-r1

wfleurant commented 8 months ago

cool i will like to move cjdns into main packages in order for future builds to compile with packages/lang/rust if ok. i can fix up for apk there?

ringtailedfox commented 8 months ago

cool i will like to move cjdns into main packages in order for future builds to compile with packages/lang/rust if ok. i can fix up for apk there?

this sounds like a good time to consider a total re-write of the cjdns packages, since cjdns segfaults when being installed and when trying to start, as noted in issue 908. the CJDNS crew have been trying to re-compile an ipk from source with me, but we're hitting a lot of stumbling blocks on this...

ecsv commented 8 months ago

Below packages that would cause trouble. Please bear in mind that PKG_RELEASE is soonish prefixed with an r and may only contain a number.

Naughty list: ...

[ ] kmod-batman-adv <LINUX_VERSION>.2023.0-r0

@aparcar the batman-adv version number is created by the OpenWrt kernel build scripts. Can you please explain us what we should do. Because I don't see any modifications in packages like mt76 or mac80211 to support apk versions - they seem to use exactly the same version number format which (according to you) we should avoid in the kmod-batman-adv package

Here some examples from my current x86-64 build (main branch):

bin/targets/x86/64/packages/kmod-crypto-hash_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-null_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-aead_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-manager_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-hmac_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-sha512_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-rng_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-seqiv_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-ctr_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-ccm_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-cmac_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-crc32c_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-gf128_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-ghash_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-crypto-gcm_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nls-base_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nls-cp437_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nls-iso8859-1_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nls-utf8_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-fs-vfat_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-hwmon-core_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-i2c-core_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-i2c-algo-bit_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-input-core_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-lib-crc-ccitt_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-lib-crc16_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-lib-crc32c_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-libphy_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-phylink_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mii_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mdio-devres_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-phy-realtek_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-r8169_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-e1000_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-pps_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-ptp_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-e1000e_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-igb_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mdio_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-ixgbe_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-tg3_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-forcedeth_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-bnx2_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-pcs-xpcs_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-stmmac-core_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-igc_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-amazon-ena_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-reject_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-reject6_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-conntrack_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-conntrack6_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-log_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-log6_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-nat_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nf-flow_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nfnetlink_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nft-core_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nft-nat_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nft-offload_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-nft-fib_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-slhc_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-ppp_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-pppox_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-pppoe_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-amd-xgbe_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-dwmac-intel_6.1.82-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-button-hotplug_6.1.82-r3_x86_64.ipk
bin/targets/x86/64/packages/kmod-cfg80211_6.1.82.6.6.15-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mac80211_6.1.82.6.6.15-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mt76-core_6.1.82.2024.04.03~1e336a85-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mt76-connac_6.1.82.2024.04.03~1e336a85-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mt7615-common_6.1.82.2024.04.03~1e336a85-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-mt7615e_6.1.82.2024.04.03~1e336a85-r1_x86_64.ipk
bin/targets/x86/64/packages/kmod-batman-adv_6.1.82.2024.1-r1_x86_64.ipk

There are no changes at all in the latest apk PRs (see https://github.com/openwrt/openwrt/pull/14918, https://github.com/openwrt/openwrt/pull/14980, ) for kernel packages in the main repo - and the main repo is responsible for the creation of these linux_version+pkg_version version numbers.

And I don't get where the .<fixup> part in the version number is coming from. Alpine linux also seems to use version numbers with only <major>.<minor>: https://pkgs.alpinelinux.org/package/edge/main/x86/batctl#

And even when assuming that 6.1.82.2024.1 would make kmod-batman-adv a "naughty package" (the adjective you've chosen): this would perfectly follow the scheme which you presented in your initial post: <major>.<minor>.<fixup>[.<sub1>...]. 6 would be the <major>, 1 is the <minor>, 82 is the <fixup>, 2024 is the <sub1> and 1 is the <sub2>. If you want a different order/format then you should have a look at include/kernel.mk (line starting with VERSION:=):

...
define KernelPackage
  NAME:=$(1)
  $(eval $(call Package/Default))
  $(eval $(call KernelPackage/Defaults))
  $(eval $(call KernelPackage/$(1)))
  $(eval $(call KernelPackage/$(1)/$(BOARD)))
  $(eval $(call KernelPackage/$(1)/$(BOARD)/$(if $(SUBTARGET),$(SUBTARGET),generic)))

  define Package/kmod-$(1)
    TITLE:=$(TITLE)
    SECTION:=kernel
    CATEGORY:=Kernel modules
    DESCRIPTION:=$(DESCRIPTION)
    EXTRA_DEPENDS:=kernel (=$(LINUX_VERSION)~$(LINUX_VERMAGIC)-r$(LINUX_RELEASE))
    VERSION:=$(LINUX_VERSION)$(if $(PKG_VERSION),.$(PKG_VERSION))-r$(if $(PKG_RELEASE),$(PKG_RELEASE),$(LINUX_RELEASE))
    PKGFLAGS:=$(PKGFLAGS)
    $(call KernelPackage/$(1))
    $(call KernelPackage/$(1)/$(BOARD))
    $(call KernelPackage/$(1)/$(BOARD)/$(if $(SUBTARGET),$(SUBTARGET),generic))
  endef
....

And I am not sure what kind of branch you've tested. bmx6 doesn't exist in main. And the batman-adv version you listed here is neither in main, openwrt-23.05 or openwrt-22.03

And you're container returns an "unauthorized" when trying to download the manifest. I couldn't find any similarly named container in the openwrt or your private "package" container registry. Wasn't this correctly published?

ecsv commented 8 months ago

You can find the version compare code at https://github.com/alpinelinux/apk-tools/blob/f9eaeb6429325eeb5a17ed771fd477be9227fe15/src/version.c#L275

It just iterates over the tokens until they are not equal anymore. So if it can be stored in apk_blob_t, it seems to be a valid version number. And since it is just a buffer with length, it doesnt seem to be require a specific number of token (or is limited to a specific number of tokens): https://github.com/alpinelinux/apk-tools/blob/f9eaeb6429325eeb5a17ed771fd477be9227fe15/src/apk_blob.h#L18

So token_first and token_next seem to be relevant: https://github.com/alpinelinux/apk-tools/blob/f9eaeb6429325eeb5a17ed771fd477be9227fe15/src/version.c#L141. It seems to require a number at the beginning and then various stuff can follow. They documented it like this:

/* Alpine version: digit{.digit}...{letter}{_suf{#}}...{~hash}{-r#} */

I didn't find (at the moment) anything which explains your requested format <major>.<minor>.<fixup>[.<sub1>...]. And still nothing which implies that the kmod-batman-adv version number (with the leading linux-version number) would be "naughty".

aparcar commented 8 months ago

Sorry of the noise, attaching the semver'ish version (2024.01) to the Linux version works just fine, I suspected the Linux version containing anything not-server and thereby breaking the version check. Right now it's 6.1.82.2024.1-r1 and with that perfectly. fine.

aparcar commented 8 months ago

@wfleurant in the meantime, I added a PR to solve the issue https://github.com/openwrt/routing/pull/1060

openwrt / routing

Preparations to switch to APK package manager #1048