Evaluate GitHub Actions as a replacement for Travis CI

[jeancochrane]

I just got beta access to GitHub Actions, a repo automation service that includes (among other things) native CI/CD.

We've been thinking about alternatives to Travis for some time now, considering that the financial health of the parent company isn't great. GitHub Actions is a strong contender because it would remove an integration layer between source control and CI/CD.

Set up a GitHub Actions pipeline for testing and deployment on https://github.com/jeancochrane/pytest-flask-sqlalchemy and evaluate its feasibility for DataMade projects. Relevant questions include:

• Pricing
• Concurrency
• Ease of setup
• Modularity across different apps
• Integration with our deployment providers (CodeDeploy, PyPi, Netlify, potentially Heroku)

[jeancochrane]

I'm feeling really good about Actions after a couple hours of testing it on pytest-flask-sqlalchemy. You can see my commit that uses it for CI here: https://github.com/jeancochrane/pytest-flask-sqlalchemy/commit/b6b9f846977e7981a0ec69d969eceb99ddee58f7

Based on my pilot project, Actions seems to give us nearly all of the most important features of Travis at a much lower price point and with native support on our source control platform. Once Actions becomes generally available on November 13, I'll be ready to put together a formal recommendation of adoption.

Below are some considerations based on my pilot project.

Features

If we narrowly consider CI, GitHub Actions offers all of the essential features that Travis offers. But since Actions is an event-driven code runner, and not simply a CI platform, it's a great deal more flexible than Travis is. Actions can run our tests, but it can also run cron tasks, scrapers, or any other code that would make sense to run periodically from our repo. Further, it can compose these operations in any order that makes sense to us.

One particularly nice feature of Actions is first-class support for containers. Actions can be defined as containers, meaning they can be composed with one another. You can also define additional services that the workflow requires in a simplified Docker Compose syntax, and Actions will wire up an internal network and expose the containers to one another. There's also native support for docker-compose workflows, which we could use to test our containerized apps.

The fact that jobs can specify hierarchical dependencies between one another, coupled with the ability to run jobs on a schedule, also makes Actions a potential solution for scrapers. I didn't consider this use-case very deeply as part of this R&D, however.

Pricing

Whereas Travis prices per number of concurrent jobs and gives you unlimited build minutes within those jobs, GitHub actions gives you 20 concurrent jobs per repository and prices per build minute. While at a certain scale this might make Actions more restrictive than Travis, the Free tier gives you 2,000 build minutes per month on private repos; according to our Travis Insights, we used about 400 build minutes last month between all our repos, including public repos and dedupeio repos. Based on these rough numbers, this means that we would essentially get CI for free.

Integrations

Integrations with Actions don't seem as robust as Travis, which is probably to be expected from a Beta service. Most notably, Actions offers email notifications out of the box, but doesn't offer a first-class Slack notification.

However, since Actions can be composed from any other Action, and since Actions can also run arbitrary code in the form of Dockerfiles, an Action can in fact integrate with any service that has an API; it just won't be as straightforward as it is on Travis. Note that there are already a number of community-built Actions for e.g. sending Slack notifications based on Workflow results (e.g. https://github.com/Ilshidur/action-slack). We could either build on one of these, maintain our own solution, or (more likely) wait until GitHub inevitably releases an officially-supported Slack integration.

A lack of integration with CodeDeploy is also a major concern. We could use https://github.com/actions/aws to send messages to the CodeDeploy API directly, but it would likely require further research to iron out the details. Note that this lack of integration will not be a problem with Heroku, since Heroku integrates directly with GitHub to determine whether CI passed before running builds.

[hancush]

tabled until actions become generally available nov. 13. @jeancochrane will draft recommendation of adoption then.

context/questions:

• we pay $140/month for travis for private repos.
• what is the future of actions? lambda competitor?
• what's timeline for migration away from travis?

[jeancochrane]

Actions are becoming generally available next week, so I'm going to pick this up again during November. Adding it back to the Backlog.

[jeancochrane]

@fgregg recommends moving this to next quarter, when the service will have stabilized and there will be a community around it.

[aszebla]

We also decided to move our applications to GitHub Actions to save some money. We described the migration process from Travis and comparison of both tools in this article, so if you are interested here you can read it: https://bravelab.io/blog/migrating-from-travisci-to-github-actions-1/

[jeancochrane]

Thanks @aszebla, this is helpful!

[jeancochrane]

I did a little bit more evaluation today in the context of https://github.com/datamade/document-search/ and https://github.com/datamade/mn-election-archive.

The big lesson is that running Actions with CodeDeploy is going to require a custom integration that I don't want to write or maintain. Because of this, I decided not to configure Actions on document-search, which is deployed on EC2.

However, I set up Actions on mn-election-archive in https://github.com/datamade/mn-election-archive/pull/58 and it was a breeze working with a Heroku app. I really like the simplicity of Actions as a test runner and the easy integration with Docker is a relief.

As a result of these tests, I would recommend that we only adopt GitHub Actions for apps deployed on Heroku (or Netlify). That would mean we would continue to support Travis for our legacy apps, and could phase out our use of it over time as we move more and more of our apps to Heroku. How does that sound to you @hancush? If it sounds good, I can draft up a recommendation along with some documentation on setting up CI with Actions.

[hancush]

I agree that maintaining a custom integration is a dealbreaker for CodeDeploy. This sounds like a reasonable recommendation to me! We may want to revisit the CodeDeploy question in six months – it doesn't seem like an uncommon problem, so maybe someone else will have written an integration by then.

[hancush]

It looks like at least a couple of GA integrations with CodeDeploy have emerged:

• https://github.com/byu-oit/github-action-codedeploy
• https://github.com/webfactory/create-aws-codedeploy-deployment

Our legacy Travis builds are glacial over in LA Metro, so I'd like to spend a day seeing if I can use one of these integrations to transition builds and deployments to GitHub Actions.

[hancush]

I'm going to move forward with https://github.com/webfactory/create-aws-codedeploy-deployment. The next step is to store our developer IAM credentials as encrypted secrets accessible by GitHub Actions. I'll start by defining them on the Metro repo. If we decide we like this approach, we should store them at the organization level and authenticate repos that need them. N.b., only admins can set organization secrets.

[fgregg]

here's a recipe for CI of a django app that needs postgres.

https://github.com/datamade/django-councilmatic/blob/2.5/.github/workflows/main.yml

[fgregg]

@hancush for a project for the courts project, I needed to set up a code deploy pathway using gitlab.

https://gitlab.com/ChicagoDataCooperative/court-terminal/-/blob/master/.gitlab-ci.yml#L13

the pattern i used was to package the assets in gitlab's CI and push them to specific s3 bucket. a code deploy instance was setup to listen to that s3 bucket and do the deployments from there

https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-s3deploy.html

maybe there's an even nicer way now, but that's at least one pathway.

[fgregg]

actually https://github.com/webfactory/create-aws-codedeploy-deployment looks pretty nice!

[hancush]

I wired this up for LA Metro and tested a staging deployment from my branch in this build.

The only downside is that running containerized tests takes about 5 minutes (with Docker layer caching, vs. about 6.5 without), while Travis builds ran in about 2:15. This still represents an improvement, because queue and boot times for Travis VMs were the real killer, lately amounting to an hour or more between build request and build run.

The other thing to note is that the CodeDeploy integration does not support deployments from tags, however this could be a positive change, as it would mean consistency with our Heroku practice of production deployments from a persistent deploy branch.

What do you think, @fgregg?

[fgregg]

i like having tagged releases as it makes clear what was deployed, but i suppose we have that in our deployment logs on AWS. I can let that go.

as for build speed, the "services" patter I used for django-councilmatic is pretty fast. https://github.com/datamade/django-councilmatic/runs/1482966504?check_suite_focus=true

[hancush]

Update GitHub Actions docs with guidance on legacy apps: Use Action, plus services pattern (above).

[hancush]

N.b., there is a tricky bug in the CodeDeploy integration that I think has been resolved but not yet rolled into a release. https://github.com/webfactory/create-aws-codedeploy-deployment/issues/5#issuecomment-743306579

Should hold off on additional migrations to this pattern until a new release is cut.

[hancush]

Update: You can pin GitHub Actions to a particular commit, so I updated Metro to reference the latest CodeDeploy action commit, which fixes the tricky bug in question. Deployments are back on track: https://github.com/datamade/la-metro-councilmatic/pull/684.

[mpdude]

👋🏼 I'm one of the authors of https://github.com/webfactory/create-aws-codedeploy-deployment/. I am really glad you found, evaluated and liked that action!

I've just released a v0.2.2 version that hopefully includes the fix you've been waiting for.

As a disclaimer: We've written the action for our own purposes, but have only used it in very limited test scenarios so far. But once we have built a bit more confidence that it works smoothly and especially that the team can handle continuous deployments without too much stress, we plan to roll it out to about 50 projects next year.

[hancush]

Once recommendation comes in, make a list of private repos to transition away from Travis so we can stop paying them.

[hancush]

Make list of private repos under DataMade and dedupeio organizations and disable Travis (inactive projects) or migrate to GH Actions (active projects).

[fgregg]

made issue here @hancush https://github.com/datamade/how-to/issues/168

[hancush]

Thank you kindly, @fgregg! Closing the book on this one.