search

datamade / how-to

📚 Doing all sorts of things, the DataMade way
MIT License
86 stars 12 forks source link

Automated image builds: Dockerhub vs. GitHub Actions #95

Closed hancush closed 2 years ago

hancush commented 4 years ago

Background

Our usual deployment platforms (Heroku and Netlify) build applications from bundled Docker images, however the Metro dashboard presented an unusual deployment configuration where the dashboard, an Airflow (Flask) app deployed to an EC2 instance, needed access to application images to run automated jobs based on scraper and Metro application code. I set up automated builds for the application dependencies in Dockerhub to meet the immediate need, and they work great!

With that said, there are some things I don't love:

  1. Dockerhub builds run in parallel to, rather than in concert with, our other CI tools.
  2. There is no indication in GitHub, our primary code repository, that images will be automatically built.
  3. It's yet another build tool in our stack.

Proposal

I'd like to compare and contrast automated builds with Dockerhub with whatever plumbing it takes to wire up a build and push an image with GitHub Actions, our tool of choice for CI.

Deliverables

This project should produce a recommendation of adoption for CI for Docker images and either update the existing GitHub Actions documentation or add a brief overview of automated builds with Dockerhub.

Timeline

I expect this will take one R&D day.

jeancochrane commented 4 years ago

I would also like to investigate GitHub packages, which offers Docker image hosting.

hancush commented 3 years ago

We should pick this up, and add a price comparison between Dockerhub and GitHub Packages, as Dockerhub is introducing a 200 pulls/6 hour period in November for free users: https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/

I don't think we'll hit this rate limit with our scraper setup, which does at most 13 pulls per hour (4 for bills, 4 for events, 4 for hourly processing, and 1 for the full scrape), which works out to 78 pulls over six hours, but we shouldn't get complacent in the assumption that they won't further decrease rate limits for free accounts moving forward.

hancush commented 2 years ago

DockerHub doesn't do this for free anymore, so we're using GitHub Actions.