search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.03k stars 3.5k forks source link

Preparations to switch to APK package manager #23706

Open aparcar opened 8 months ago

aparcar commented 8 months ago

Hi all, some fellow developers and me worked for some time now on making APK the new package manager for OpenWrt, replacing the unmaintained OPKG fork we've been using for the longest time.

APK is actively developed and used in multiple other distributions, e.g. Alpine Linux 🎉

While there is till some work ahead, I'd like to prepare everyone who maintains a package to verify that the PKG_VERSION follows Semantic Versioning <major>.<minor>.<fixup>[.<sub1>...]. APK uses a deterministic algorithm to compare versions and does not like random strings, except a valid hash prefixed with a ~.

If in doubt, please use the Docker container below to verify the used version is valid:

docker run --rm -it ghcr.io/aparcar/apk-valid-version <VERSION> [<VERSION> ...]

It will print whatever version is not valid, if you get a zero exit code, you're fine.

Please feel free to reach out for assistance and have a look at the core migration of versions.

aparcar commented 8 months ago

Below packages that would cause trouble. Please bear in mind that PKG_RELEASE is soonish prefixed with an r and may only contain a number.

Naughty list:

brada4 commented 8 months ago

isc-dhcp is EOL https://www.isc.org/dhcp/

BKPepe commented 8 months ago

Dear package maintainers,

There is no need to open individual pull requests for your packages, you are maintaining. That will be really time-consuming for you and for us. We will do it here exactly the same as it was done in the main repo via treewide commit.

stokito commented 8 months ago

I didn't know that the OpenWrt is going to switch to APK. Here is more details on this https://forum.openwrt.org/t/rfc-announcing-apkwrt/120096

pesa1234 commented 8 months ago

Dear, I've update ksmbd https://github.com/openwrt/packages/pull/23750, I'm not maintainer but if I can why don't help? Thanks guys

stangri commented 8 months ago

@aparcar Just a heads up, most of my packages either:

I believe I have fixed most of those, but I may have to submit further PRs if needed.

Also, the https-dns-proxy should be fixed with the latest PR.

PS. If the PR needs to be made against packages/luci repo for reasons other than APK migration, should I refrain from moving to the new release syntax?

neheb commented 7 months ago

libmad was fixed.

pesa1234 commented 7 months ago

ksmbd-* fixed

neheb commented 7 months ago

I noticed mtd-rw has a bad version too. https://github.com/openwrt/packages/pull/23977 fixes.

ErwanMAS commented 6 months ago

@aparcar 

root@debian:~# docker run --rm -it ghcr.io/aparcar/apk-valid-version
Unable to find image 'ghcr.io/aparcar/apk-valid-version:latest' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/aparcar/apk-valid-version/manifests/latest": unauthorized.
See 'docker run --help'.
stokito commented 6 months ago

The sslh migration (just as a version update) https://github.com/openwrt/packages/pull/24192 Its author confirmed that he will now always use a semantic versioning

ttc0419 commented 5 months ago

Quick question, is 24x going to use apk?

aparcar commented 5 months ago

Quick question, is 24x going to use apk?

No final decision on this.

stangri commented 4 months ago

Quick question, is 24x going to use apk?

No final decision on this.

If 24.xx is NOT going to use APK by default, would the 24.xx building tools (toolchain, SDK, CI) still prepend r to the PKG_RELEASE?

aparcar commented 3 months ago

@openwrt/packages-write I kindly ask every maintainer to check the failing package builds here and see if the PKG_VERSION need some adoption: https://buildbot.aparcar.org/faillogs/x86_64/packages/

hnyman commented 3 months ago

@aparcar In addition of versioning, there appears to be also dependency trouble. Some packages depend explicitly on opkg, and building an actual firmware will fail if apk is selected. Apk also finds more conflicts that opkg.

I built yesterday an APK enabled firmware, and had to remove one package (openssh sftp server) due to the versioning reasons, like expected. But then at image finalization I ran into conflicts with e.g. @dibdot popular 'adblock' which explicitly depends on opkg.
Similarly, some packages depend on uclient-fetch,, which apk finds to be in conflict with GNU wget, unlike opkg does.

ErwanMAS commented 3 months ago

@aparcar

root@debian:~# docker run --rm -it ghcr.io/aparcar/apk-valid-version
Unable to find image 'ghcr.io/aparcar/apk-valid-version:latest' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/aparcar/apk-valid-version/manifests/latest": unauthorized.
See 'docker run --help'.

@aparcar i want to try your docker image and i have this error ^

dibdot commented 3 months ago

But then at image finalization I ran into conflicts with e.g. @dibdot popular 'adblock' which explicitly depends on opkg.

@hnyman I'll remove the opkg dependency in adblock with the next minor update - thanks for heads up!

Edit: the opkg dependency has been removed in adblock-4.2.2 (see 34db79bcd584f2da9a64dd4c1e84f138e3e4f70b)

Payne-X6 commented 3 months ago

@aparcar What to do when apk compatible version format opkg compares as an older version than the original? See libedit.

nxhack commented 2 months ago

node, node-npm: I have sent a pull request.

Update: merged

cotequeiroz commented 2 months ago

libopenssl-afalg_sync: fixed by #24897

E021ntox commented 1 month ago

Will APK be integrated into 24.10?

neheb commented 1 month ago

Unlikely unfortunately.

E021ntox commented 3 weeks ago

Good good news. https://lists.openwrt.org/pipermail/openwrt-devel/2024-October/043349.html

Neustradamus commented 2 weeks ago

For the moment, it is not sure for 24.10.x?

E021ntox commented 1 week ago

State of APK package manger integration - switch on Monday! http://lists.openwrt.org/pipermail/openwrt-devel/2024-November/043378.html

hnyman commented 1 week ago

List of still failing packages can be seen from the test/staging apk buildbot faillogs: https://downloads.staging.openwrt.org/snapshots/faillogs/

only a few architectures have been compiled so far, but seems that e.g. ddns-scripts, nginx, ntfs-3g, many perl packages etc. still fail due to versioning reasons.

vortexilation commented 1 week ago

Does anyone have a clue about perl-file-next or perl related packages?

ERROR: info field 'version' has invalid value: package version is invalid
ERROR: failed to create package: /home/user/works/openwrt/bin/packages/x86_64/packages/perl-file-next-1.18+perl5.40-r1.apk: package version is invalid

Looking at for example perl-file-next Makefile PKG_VERSION, i don't see anything wrong with it : 

PKG_NAME:=perl-file-next
PKG_VERSION:=1.18
PKG_RELEASE:=1

What would be the fixes then?.

aparcar commented 1 week ago

Seems like you attach perl5.40 to the version itself, which doesn't fly. How about something like perl-file-next-1.18.5.40-r1? This is what we did for some kernel packages, first the package version attached by the Kernel. You may not any arbitrary strings to the version anymore.

1715173329 commented 1 week ago

guess we need change https://github.com/openwrt/packages/blob/ca503cc4054d9a13558c7b552886a6dba359a0eb/lang/perl/perlmod.mk#L11 to PKG_VERSION:=$(PKG_VERSION).$(PERL_VERSION2)

vortexilation commented 1 week ago

@aparcar Thank you, will be helpful for fixing other packages version related things.

@1715173329 Compiled fine now, thanks, I think you should open a PR as using APK is becoming default now. These will be fixed all perl APK related problems as listed above.

PussAzuki commented 1 week ago

I found that when compiling nginx-ssl I get an error with the following output

"depends:nginx-ssl-util>=1.5-1 libc libopenssl3 libpthread libpcre2 nginx-ssl-util zlib"

How to fix nginx makefile?

hnyman commented 1 week ago

How to fix nginx makefile?

EXTRA_DEPENDS:=nginx-ssl-util (>=1.5-1)

By removing the extra version comparison there. @Ansuel probably thinks about better fix, but that should be enough for your own use.

(Not sure if actually just "1.5-r1" would be enough there, as old "1.5-1" is not accepted by apk)

Ra2-IFV commented 1 week ago

I see a lot of package use this:

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.xz

Then PKG_VERSION must match upstream version, this breaks apk support if upstream version use other naming syntax. Do we have a solution for this?

Ra2-IFV commented 1 week ago

I found that when compiling nginx-ssl I get an error with the following output

"depends:nginx-ssl-util>=1.5-1 libc libopenssl3 libpthread libpcre2 nginx-ssl-util zlib"

How to fix nginx makefile?

25335

hnyman commented 1 week ago

PKG_VERSION must match upstream version, this breaks apk support if upstream version use other naming syntax. Do we have a solution for this?

Some examples of solutions, mainly involving extra variables in Makefile to differentiate PKG_VERSION and the name needed for upstream file (and possibly for compilation PKG_BUILD_DIR):

Ra2-IFV commented 1 week ago

It works, but it would be better if there is a standard for this

hnyman commented 1 week ago

If you check the currently failing packages in faillogs, you notice that there is substantial variation to the upstream names. There is no single cure. https://downloads.openwrt.org/snapshots/faillogs/aarch64_cortex-a53/packages/

hnyman commented 5 days ago

The list of currently failing packages in the packages feed is already below 10 packages.

https://downloads.openwrt.org/snapshots/faillogs/x86_64/packages/

debootstrap/ hfsprogs/ kafs-client/ micropython-lib/ netifyd/ oci-runtime-tools/ python-dateutil/ qcsuper/ sox/

( and also lua-curl-v3 and netdiscover, but there is already a PR for them )

Ansuel commented 5 days ago

@hnyman always dreap to have 0 faillogs there... at least for some targets :(

Ansuel commented 5 days ago

This error looks interesting

https://downloads.openwrt.org/snapshots/faillogs/x86_64/telephony/pcapsipdump/compile.txt

Ansuel commented 5 days ago

/home/ansuel/openwrt-ansuel/staging_dir/host/bin/fakeroot /home/ansuel/openwrt-ansuel/staging_dir/host/bin/apk mkpkg --info "name:pcapsipdump" --info "version:2020.03.03~157-r1" --info "description:pcapsipdump is a tool for dumping SIP sessions (plus RTP traffic, if available) to disk in a fashion similar to "tcpdump -w" (format is exactly the same), but one file per SIP session (even if there are thousands of concurrent SIP sessions)." --info "arch:aarch64_cortex-a53" --info "license:GPL-2.0+" --info "origin:feeds/telephony/net/pcapsipdump" --info "url:http://sourceforge.net/projects/pcapsipdump/" --info "maintainer:" --info "provides:" --script "post-install:/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/apk-aarch64_cortex-a53/pcapsipdump/post-install" --script "pre-deinstall:/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/apk-aarch64_cortex-a53/pcapsipdump/pre-deinstall" --info "depends:libc libstdcpp6 libpcap1" --files "/home/ansuel/openwrt-ansuel/build_dir/target-aarch64_cortex-a53_musl/pcapsipdump-2020.03.03~157/ipkg-aarch64_cortex-a53/pcapsipdump" --output "/home/ansuel/openwrt-ansuel/bin/packages/aarch64_cortex-a53/telephony/pcapsipdump-2020.03.03~157-r1.apk" --sign "/home/ansuel/openwrt-ansuel/private-key.pem"

Ok we need to escape the description...

hnyman commented 5 days ago

Yeah the description handling shout be fixed. Not sure if it should be escaping or sanitising... I am so old-school, that having possible string delimiters (double quotes) in the field sounds error-prone in the long run.

Ansuel commented 5 days ago

@hnyman I sanitized the package description and also pushed a commit that escape them just to handle anything that is not officially supported.

hnyman commented 3 hours ago

All packages in main/master should now be fixed for apk versioning. In the last two weeks I have patched the last remaining dozen failing packages or so.

Buildbot doesn't currently show any apk related failures for main OpenWrt, LuCI, packages & telephony. (once the packages I patched just few minutes ago are built there next time)

In routing feed there is still PR https://github.com/openwrt/routing/pull/1089 waiting for @aparcar @Ansuel or somebody else with rights to the routing feed repo.

I have backported my own commits to 24.10, but I suspect that some other packages are still failing there., as fixes have not been backported. (Note: Buildbot does not yet show failures there, as opkg is still currently the default in 24.10.)