Consider the options migrating from Travis CI .ORG

[jimklimov]

According to https://travis-ci.community/t/builds-hang-in-queued-state/10250/24 and https://travis-ci.community/t/org-to-com-migration-deadline/10260 the Travis CI team had poorly communicated that the free .ORG service will cease end of this year, and projects should move to .COM. Also there is https://docs.travis-ci.com/user/migrate/open-source-repository-migration#frequently-asked-questions and https://docs.travis-ci.com/user/billing-overview/, and the timeline and discussion of options at https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works

• From the latter, the point on "I don't have enough time in my day to send off emails every few [days|weeks|months] requesting extra build credits so I can continue maintaining my open source projects." sounds all too familiar and painful...

In fact, like many others in those threads, I only found out by looking for reasons of the recent lags while the status says all systems are green. Further in the first linked thread, there are some concerns about how much and how easily FOSS projects can build in the new setup, and it is even unclear whether their free allotment is a single-time deal or a monthly quota. If a quota is overflowed, it may be possible to ask for a bump which would likely mean regular bureaucracy and lag in PR tests to live through a month, and/or possibly a more active need for sponsorship from users and companies to pay for a plan, month to month.

Alternate options include going to GitHub Actions, or spinning up free VMs in the cloud providers (some provide a way to have free low-profile services; not sure if running a builder occupying a CPU core 24/7 counts as fair use for that sort of service), or buying some VMs or physical builders, hosted at providers or at home for the community members - but again can become a matter of sponsorship in the end. On the up-side, dedicated workers can be used with varied OSes not provided by common cloud CI.

Considering recent work to build in NUT as diverse environments as we could grab (#823, #844) the availability of multiple options in Travis was certainly helpful to bring code up into shape against standards and OSes. Thanks to them for providing as much as they did in the free .ORG service, in any case.

But the future default matrix of tests, whether on Travis CI .COM or on some other solution, likely would have to be reduced to conserve their resources and to speed our PR CI turnaround:

• Spanning just an oldest available compiler with C99+__EXTENSIONS__ and gnu99, and newest compiler with newest standard, with no intermediate steps. Other related savings could include less runs with e.g. same/similar versions of same compiler in different OSes (assuming clang or gcc of same revision returns same standards-related warnings on different hosts). On the downside the contributors have less visibility on breakage they might introduce for cases they can't test locally, and the NUT team and community can't be as sure about which systems we support well and where we can improve.
• We could also clump together some of the smaller tests, e.g. shell scripts validity, to reduce the overhead of deployment and configuration vs. actual tests (Travis CI ideology did promote spinning up separate test-cases and so builders).
• Preferably the different non-default ways of building the tests should still be defined, just optional, so something like a "monthly build" could enable everything to check that we remain in shape over time but not spend resources for every build to check this is the case.

[jimklimov]

Over the past few weeks, I've sent several e-mails to Travis support asking how we can go forward or get an OSS credits allotment from them, but so far had no replies. This seems in line to experience of most posters on the Travis forum, and in contrary to loud statements about supporting OSS for free forever. Still, hope lingers that they are just overwhelmed with the migration and changing the rules of the game, and that things might stabilize.

A CI account was switched to the .COM offering, quickly ate through the pre-assigned 10k credits, and stopped building. These did not replenish since, so far...

[jimklimov]

https://www.oracle.com/cloud/free/#always-free offers two compute VMs for infrastructure needs, with "1/8 OCPU and 1GB RAM" which is rather meagre by common standards, but might be enough to run builders for uncommon OSes not provided in other free offerings (if the VMs can be installed into anything of our choice -- seems to be possible per https://docs.oracle.com/en-us/iaas/Content/Compute/References/images.htm => https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/managingcustomimages.htm#Managing_Custom_Images and https://docs.oracle.com/en-us/iaas/Content/Compute/References/bringyourownimage.htm docs).

[jimklimov]

One more option was mentioned recently, https://fosshost.org/about

Those open source projects eligible for our services, typically apply for a virtual private server (VPS). This is typically provided with the following specification, but can be increased, upon request.

6 vCPU
4GB Memory
200GB Storage
IPv4 / IPv6 connectivity
Full Remote SSH Access

We support most operating systems including CentOS, Debian, Ubuntu, Gentoo, ArchLinux, Fedora and FreeBSD. We support custom OS.

• https://docs.fosshost.org/en/home/application
• https://docs.fosshost.org/en/home/what-do-we-expect
• https://docs.fosshost.org/en/home/virtual-private-server
• https://docs.fosshost.org/en/home/arm64-as-a-service

[jimklimov]

Great thanks to fosshost.org, they have provided us some virtual machines where we can build up a CI farm over a diverse choice of target operating systems to take over from Travis (which so far still happens to work overtime in the free Org offering), so I'd get to that as soon as possible.

[jimklimov]

As announced on the mailing list at https://alioth-lists.debian.net/pipermail/nut-upsuser/2021-August/012484.html the new https://ci.networkupstools.org/ diverse farm on VMs from Fosshost went live this weekend, and is currently building some legacy milestones from the master branch so we have a record of code quality changes over recent past (2.7.4 release and different PRs related to #823 "fightwarn" effort).

The new solution uses a custom-built https://github.com/networkupstools/jenkins-dynamatrix/ library to arrange diverse CI builds driven by capabilities of currently present build agents.

[jimklimov]

Over time hope to document the solution with PRs like #1094 and maybe some blog about the backs-and-forths and trade-offs that the solution required.

[jimklimov]

Another viable option recently recognized is AppVeyor which offers one simultaneous build for public/FOSS repositories, and unlike CircleCI it does not seem limited by CPU-Minutes per month given for free.

According to https://www.appveyor.com/docs/custom-build-images/ it is possible to define pre-built OS images (pre-install packages) and so speed up actual builds. But this is at the moment a "Trial" feature leading to page on pricing, more details are not known.

[jimklimov]

Per https://dev.to/itnext/the-ultimate-free-ci-cd-for-your-open-source-projects-3bkd Azure is less constrained for FOSS (or at least was in 2019, before GitHub Actions were rolled out)

[jimklimov]

There is also https://cirrus-ci.org/

Build your Open Source projects on Linux, Windows, macOS and FreeBSD for free, including ARM architecture!

Hmmm... For a practical example, can look at https://github.com/collectd/collectd/blob/main/.cirrus.yml

[jimklimov]

https://stefanscherer.github.io/use-appveyor-to-build-multi-arch-docker-image/

One worker manifest for different CPU platforms and OSes, neat. Perhaps it could help NUT contributors and testers spin up a build by having everything pre-installed... Would not be a best-fit build for their OS though.