[Feature request] adding auto-tests stuff to termux packages.

[twaik]

There are a lot of packages and it is pretty much hard to perform tests in the case if you do not know what package do. My idea is to add some additional file or function of build.sh which will describe exact steps needed to test package or its dependencies. Let's say we have libssh2 package which has nmap, sheldon, libcurl and apt (because of libcurl). In this case it will be something like this:

termux_step_auto_test() {
    pkg up # to check if apt still works
    pkg install nmap sheldon
    nmap -v
    sheldon
}

These commands should be launched on device manually or automatically via ssh <device> "$(source build.sh; declare -f termux_step_auto_test); termux_step_auto_test" after successful package installation. Also this approach can be used to check if package works in packages workflow while building x86_64 version on package in conjunction with termux-docker. Also it can help us use package's own tests for integrity check and avoid some issues like #20490, #21144 or #21845.

This will let me and some other maintainers process and test updates faster and more reliably.

[Grimler91]

This would be awesome! I think we could even test on real hardware, I have (too many) old devices laying around that could potentially be used in a test farm. Will be a few months before this can become reality though, I don't have good internet at the moment.

Maybe it would be clearer to have the test script(s) in a test/ subfolder (like packages/libssh2/test/), similar to what some packages do for the deb scripts

[TomJo2000]

Big +1 on having an (optional) tests/ subdirectory in package directory for a package. That way the test runner can just iterate over the scripts in that directory.

[twaik]

I have (too many) old devices laying around that could potentially be used in a test farm

In most cases test will be done on maintainers' or contributor's machines, I do not think you want to give access to your devices (even temporary) to numerous 3rd parties, right? Probably it will be fine to use termux-docker with an option to enable QEMU emulation for aarch64/armv7 AND allow people to list their devices accessible with SSH in ~/.termuxrc.

[twaik]

In addition to what I said above, we could enable mandatory tests in CI only for x86_64 for regular packages and for all archs (with QEMU) for critical/bootstrap packages to not miss anything and avoid "CANNOT LINK EXECUTABLE" errors.

[SamB]

It would be cool if we could do this in a chroot/container that cannot see the builder's /usr/bin so we can catch things like, oh, $PREFIX/bin/xmlto hardcoding paths to tools found on the build system rather than in Termux.

(I'm trying to figure out a way to force the configure script not to do that.)

/bin is obviously trickier since that actually has a few things in it on Android.

[twaik]

@SamB https://github.com/termux/termux-docker provides termux-only environment, without access to host's or builder's /usr/bin...

[Grimler91]

I have (too many) old devices laying around that could potentially be used in a test farm

In most cases test will be done on maintainers' or contributor's machines, I do not think you want to give access to your devices (even temporary) to numerous 3rd parties, right?

I believe I can setup a separate network and isolate those devices, so letting the CI of termux-packages access them with the help of secrets in termux-packages should be fine I think. Should only be run for PRs, after a maintainer has pressed that "approve workflow run" button for first time contributors.

Probably it will be fine to use termux-docker with an option to enable QEMU emulation for aarch64/armv7 AND allow people to list their devices accessible with SSH in ~/.termuxrc.

Yeah, qemu+termux-docker would be more convenient, then anyone can run it locally or during development in their own branch! Probably somethings (termux-api for example) needs actual hardware though, so having both would be nice. Yeah,

[robertkirkman]

qemu+termux-docker

keep an eye out in the near future for official free-tier aarch64 Github Actions runners which might allow removal of the dependency on qemu from termux-docker for efficient integration-testing of aarch64 packages?

[twaik]

We can not run official android ndk on aarch64 hosts so we can not build packages using aarch64 runners. Probably we can add separate workflow to the chain but it will require archiving the whole build directory and passing it to the next workflow, that might not be an option.

[robertkirkman]

The idea I had is to heavily use the termux-docker container in order to build in the "$TERMUX_ON_DEVICE_BUILD" = "true" codepath; however you are correct that it would require a very large amount of work in order to reliably compile every single package successfully in that codepath. I will make sure to fix every package for the cross-compiling codepath first before working on that codepath.