Continuous Integration (CI)

Coming from #45:

0) we should test both aarch32 and aarch64, 1) it would be nice to avoid qemu-user-static with its incomplete support and 2) other projects in beehive-lab regularly use up 100% of the included services

I'm not sure it would be that useful to have CI that regularly stops working and it also means we should avoid inefficient solutions like pulling in mambo-vm which would have been an easy fix. Self-hosted runners seem to be recommended against for public repositories because of security concerns. Any ideas?

Regarding CI, it looks like Travis CI supports ARM natively and they have a free plan for open source projects. It might work better for us.

First, I should clarify that my main motivation to suggest using CI is NOT to test MAMBO. I know that it is regularly built and thoroughly tested on multiple different platforms. Instead, I expect CI to make interaction with users easier:

It is a showcase for users unfamiliar with MAMBO. They can see not only the codebase, but also how it is actually used, which output is produced, which are the limitations, the performance, etc.
Users can fork, modify and test MAMBO using a browser only. Not an ideal workflow, but useful for really quick tests/proofs.
It provides a reproducible environment for users to report bugs/support requests. All they need to do is create a branch and add a step to retrieve/build their tricky binary. In fact, it is possible to implement an Action that retrieves the body of github issues (issue-runner).

Regarding QEMU, as you already know (beehive-lab/mambo-vm), there are qemu-system and qemu-user variants, and the support in the later is incomplete. However, combining qemu-user and Docker allows some interesting use cases:

Since 2017, docker-for-win, Docker Desktop for Windews, Docker Desktop, name it whatever... has built-in qemu-user for Arm targets: https://github.com/dbhi/qus/blob/master/docs/refs.md#linuxkitbinfmt--dockerbinfmt. Hence, users can execute the same arm32v7/ubuntu:bionic or arm64v8/ubuntu:bionic docker container on either Windows on amd64, Linux on amd64, or Linux on armhf, aarch32 or aarch64. We have tested this on several workstations, on Pine64, Rock960 and Merlin.
As mentioned in dbhi/docker, MAMBO can be built and tested on e.g. an arm32v7/ubuntu:bionic Docker container (on Windows or Linux), and the resulting artifact can be copied to a ZYNQ device running PYNQ. Note that no cross-compilation is involved. The container is used on amd64 as it would on an Arm SBC.
It potentially allows to use services such as gitpod or play-with-docker to tinker with MAMBO. Note that, actually, it does not work on play-with-docker. I have not tested on gitpod. However, at some point similar services might exist which support running docker containers for Arm.

Hence, even if all tests cannot be executed using Docker + qemu-user (specially due to missing signal support), I think it is still valuable, since MAMBO can be built and several plugins do work.

we should test both aarch32 and aarch64

Agree.

other projects in beehive-lab regularly use up 100% of the included services

Execution time of MAMBO's CI would be mostly negligible; it's so fast. Anyway, GHA workflows don't need to be executed for each push. They can be scheduled, executed only on tagged commits, whatever. I belive that it is important to have the infrastructure set up. Then, it is trivial for any user to fork and change CI triggering conditions. We can explicitly document how to do it.

I'm not sure it would be that useful to have CI that regularly stops working

My proposal is that only failing builds of MAMBO produce a failing run. Tests are executed, but results do not change the state of the run.

Of course, I think it would be much better to have any other complementary solution which allows to properly evaluate all the tests. Having some "test runner" that provides a nicer output would also be interesting.

it also means we should avoid inefficient solutions like pulling in mambo-vm which would have been an easy fix.

Both GHA and Travis support caching. Hence, pulling in mambo-vm should not be much of an issue. It would only be pulled from "external" resources when it is updated. Furthermore:

mambo-vm can be built/updated and/or released as a conditional step in the same workflow, or in a separate workflow in its own repo.
https://github.com/features/packages is free for open source public projects.

From a performance point of view, I don't think there is much difference between retrieving a release asset from a repo or a package from a registry.

Anyway, the main reason I didn't use mambo-vm is because I don't know how to start the VM, share a folder from the host and execute some script non-interactively. Does it need to be done with scp + ssh?

Self-hosted runners seem to be recommended against for public repositories because of security concerns. Any ideas?

Yes. However, I think that self-hosted runners can be useful for your own use case. Since you have multiple boards, you can use GitHub Actions to trigger a test in all of them at the same time, and have the results displayed. Optionally, assets can be uploaded. If you limit self-hosted runners to workflows that are not triggered by PRs, you should be good. I believe that forks cannot use self-hosted runners of the parent.

Naturally, you can achieve the same result without GHA, using a self-hosted GitLab o Jenkins in some spare workstation; which I would recommend.

Regarding CI, it looks like Travis CI supports ARM natively and they have a free plan for open source projects. It might work better for us.

I'd like to provide working and ready-to-use solutions for users on amd64 laptops or wokstations which don't have an ARM board at hand. I'm thinking about trips, tutorials, workshops, etc.

The only issue with Travis is that it is an external service:

On the one hand, users need to register and enable it. This is so fast and easy, but it needs to be done. I don't know whether it changed, but some time ago Travis requested a lot of permissions. Using a machine account is recommended.
- Releasing assets requires generating a token. Conversely, in GHA a default token is provided which has the scope limited to the repo.
On the other hand, during the last couple of years Microsoft bought GitHub, Actions were launched and I think that Travis was bought. I don't know whether these had any direct impact, but I've seen Travis' integration with GitHub getting worse. When workflows need to interact with GitHub's API to clone, get releases, upload artifacts... I get errors and denials regularly. Restarting the jobs fixes it, but it requires some care.

Hence, I'd like to keep qemu-user (which is just two lines), I'd like to add mambo-vm (I can take care of caching, etc. should you help with using it non-interactively) and we can use Travis for completeness too. Tests in GHA + mambo-vm and Travis should be equivalent, while tests in GHA + qemu-user would be a subset.

beehive-lab / mambo

Continuous Integration (CI) #46