Consider GitHub Actions for CI

[nolar]

Originally proposed in #551.

Consider GitHub Actions for CI, compare with Travis CI. Make a proof-of-concept for all the CI steps: static analysis, unit-tests (all python versions), integration tests (selected k8s versions via k3d/k3s), lengthy nightly tests (all k8s versions via minikube). Ensure the feedback time is ≈5 mins per push as with Travis.

The questions to answer: What would be the advantages? Better developer experience?

---

Join the beta programme, and consider Travis's own move to GitHub Apps:

• https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com/
• https://blog.travis-ci.com/2018-05-02-open-source-projects-on-travis-ci-com-with-github-apps

Would it be the same advantages/disadvatnages after their migration?

---

Beware the limit of 2000 free build-minutes per month in GutHub Actions: https://gabrieltanner.org/blog/an-introduction-to-github-actions — does Kopf fit into it?

Currently, one full build is ≈25 build-mins. One nightly build is ≈90 build-mins — i.e. 90*30 = 2700 build-mins/month — already above the free plan.

Travis, for comparison, has its mission to keep it free for open-source projects (though, I would donate a bit).

[Jc2k]

I have recently migrated a bunch of my personal projects to use it and thought I would offer my own perspective on this. Not to try and convince you, but just to offer my own experience with both of them. I have used travis for many many years, have been using gitlab ci for a good few years and have been trying GitHub Actions for a couple of months. And before that I was a contributor to Buildbot, doing a lot of work with their VM-on-demand functionality and was the original author of the shim that allowed buildbot to be configured by a .travis.yml.

I was initially skeptical but once i started i found it quite easy to port over my stuff. Before the syntax switched to YAML i was definitely not a fan. I do prefer travis and gitlab config still, but in practice they are quite similar when you get going.

I feel like my builds are quite a bit faster. The average build on travis for one of my projects was about 7-9 minutes. It looks like on GitHub actions its now 3 minutes. The GitHub actions build is doing 2x tests on macOS and 2 Windows builds it didn't do at all previously.

I think it's definitely an improvement if you want to make parts of your build process re-usable.

What was most useful for me was the macOS and Windows CI (2 of the projects are used by Home Assistant and in theory they need to be cross platform). This was a doddle and considerably faster than macOS on Travis. While this was great for me, I don't think it's a valid reason for Kopf?

What was interesting but I haven't experienced in practice is the integration you can do with the PR process. Flake8 results can be annotated on a PR or commit. Right now you can see one here (i don't know if the annotation will get hidden by a later build so it might be gone by the time you read this). It can do the same for tests. This is interesting, but not interesting enough to switch by itself. And it might be possible to do with other CI systems these days.

AIUI it is actually completely free for Open source projects. The 2000 minute limit is for private repositories, not public ones. For an example of a "stress test", check out Home Assistant. Their CI was running well over 40 minutes. They've actually split their pytest invocations into 4 and they are able to run them in parallel on GitHub actions more than on travis so the time to feedback on your PR is now down to 15minutes. 11,000 tests are split into 4 groups and each batch is run for py 3.7 and 3.8. Each one of those runs in parallel. So despite taking 10 minutes each the entire suite is still only 15 minutes. They couldn't do that on Travis because of limites on numbers of parallel tasks. Having a quick skim at their build history, they are doing 8 runs like that an hour. Kopf would fit based on that!

Other than that, both get the job done. I have switched a bunch of things so I think I clearly prefer GHA. But if you don't need the parallelisation speed ups or cross platform support i wouldn't say it was a priority to switch, really.

[nolar]

@Jc2k Thanks for sharing your experience. This is valuable. I would try GH Actions anyway in some nearest future, for sure. And will see how it works, and how it feels. Better reporting of statuses/linting/test failures directly in the code could be convenient indeed.

[Jc2k]

Glad it was helpful. I really appreciate the work you are doing on Kopf, so I hope to contribute more practically again soon.

[eshepelyuk]

My 5 cents.

1. speed of builds, it seems GH actions are faster than TravisCI, looks like they bootstrap their containers \ vm faster than TravisCI
2. pipeline syntax is more idiomatic (like Python tries to be) and easier to use, would like to underline following
   • possibility to upload artifacts from step \ job
   • share artifacts between jobs
   • clean and eplicit declaration of job steps
   • possibility to declare step outputs
   • use outputs between steps and jobs

[Jc2k]

Just saw this in my RSS feed:

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

If i understand this correctly, all "free" travis plans will migrate to free trials (that last for a year or until all trial credits are used). There is a mechanism to apply for OSS credits - "We will be offering an allotment of OSS minutes that will be reviewed and allocated on a case by case basis.". The phrasing - that they are "allocating" from an "allotment" - makes me think that there is not an unlimited capacity here and that they may start saying no.

[nolar]

@Jc2k As I read through the doc, it seems that they are talking about their commercial offering (.com). Kopf is built on their open-source system (.org) — which is a free system dedicated to open-source projects limited to max 5 concurrent builds. And they distinguish the two (.com vs .org) in their docs clearly and often.

Kopf is probably not affected. Though, I admit, the article is not clear about their .org OSS offerings. I will clarify that.

[Jc2k]

That was originally the case, though I was under the impression that travis-ci.org was getting shut down (https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom). Have they revised their plans?

[Jc2k]

From https://twitter.com/travisci/status/1322118491927957505:

We’re in the process of migrating projects from http://travis-ci.org to http://travis-ci.com - you can find out more about how this might affect your builds here:

Which links here - https://docs.travis-ci.com/user/migrate/open-source-repository-migration#frequently-asked-questions

And in the FAQ there:

In an effort to ensure that all of our users - whether you build open-source, public or private repositories - receive regular feature updates, security patches and UX/UI enhancements, we are announcing that travis-ci.org will be officially closed down completely no later than December 31st, 2020, allowing us to focus all our efforts on bringing new features and fixes to travis-ci.com and all of our awesome users like yourself on the travis-ci.com domain.

[nolar]

They have answered:

At Travis CI we do support Open-Source projects and you are probably part of this community as well.

You may already know that we have released new pricing models [1][2] to provide fair usage and balance between free usage and paid users. You may also see that your account plan has been changed as well. We have provided 10,000 credits for those who were previously using Trial builds, this is equivalent to a refresh of your Trial builds count, which can be used for both private/public projects.

We do think that this amount of credit will be sufficient for your usage to trial the product. For further usage, you’d need to either choose a plan going forward or submit a request to be considered for our Open Source Plan that is in the works. In the meantime, please don't hesitate to contact us with questions and concerns, we will happily assist you with this.

I hope this clarifies the issue on your end as well, please let us know if you need further information.

Regards,

[1] https://docs.travis-ci.com/user/billing-overview/ [2] https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

However, Kopf still sits in .org. I don't know when exactly it will be migrated to .com. "Migration is not possible for this account" — it says when I try to migrate manually. I'd better wait-1.

The only plans I see, are the concurrency-based plans. Usage-based plans are not on the list. I'd better wait-2.

Because of this, I cannot test how the credits actually work and how they are spent. Theoretically, this will be 10'000 credits per month, 10 credits/build-min (Linux), i.e. 1000 build-minutes. Kopf has spent 3800 build-minutes in the past month, and it was not the most active month of development. Either I will buy those credits (I didn't get what's the price), or move to other free platforms.

Anyway, I see no reason to urgently make any moves now, before the things get clear. I'm actually in favour of supporting a local producer and employer (Travis) by some little hobby money rather than giving that money (or praise) to a big corp far away behind the ocean ~with no local offices~ (GitHub) — as long as the features and benefits are comparable, of course.

But thanks for bringing that topic here — at least, I will be aware of the changes that are happening somewhere in the background.

[nolar]

This graph from https://www.traviscistatus.com/#month is worth being attached here for history before the pre-change data (non-flat heartbeat) is not gone:

It shows that all open-source projects are capped at 500 builds, with a backlog peaking to 6000-7000 builds every workday.

As an example, #573 was pushed 16:46, started at 17:45, i.e. 1 hour later — https://travis-ci.org/github/nolar/kopf/builds/742974926. Worth mentioning is that this 1-hour wait time is happening when the open-source backlog is at its lowest daily point, not even at the peak.

This makes Kopf's PR builds delayed by tens of minutes or even hours, essentially nullifying all effort made to speed up the PR feedback time in #536 & co (was ≈5-6 mins since the push).

I've initiated the migration from Travis CI "org" to Travis CI "com" with the free open-source plan mentioned above. Let's see how ti will behave there.

Getting any data on GitHub Actions waiting times and a similar backlog graph would be helpful.

UPD: Though the delays did not occur in the next few hours, and all builds were started immediately. Maybe it was an unrelated incident.

[nolar]

For the record:

And I could not find how to top-up the credits.

Hence, switched to GitHub Actions for the CI builds: #582

[nolar]

Done: the repo is now running on GitHub Actions.