App containerisation and Docker

[sofer]

Currently, students appear to be deploying to GitHub Pages, Heroku and Netlify, depending on context. Heroku is mentioned in in the notes for the Week 5 project. Netlify is not mentioned anywhere. It could perhaps usefully be added as early as Week 2 as a way to hide API keys (where they are currently mentioned for the first and only time).

In the second half of the programme, I would like to advocate for introducing students to containerisation through deploying Docker containers to GCP. It's conceptually simple and the basics can be picked up in under a day. It's a significant gap in the curriculum and will open up junior DevOps roles to our graduates of which there are many currently available.

A rough and ready intro to Docker and GCS.

[oliverjam]

I agree we should be stricter about getting their projects deployed each week. We can get them deploying frontends to Netlify really early on. I'm not sure this will actually help with hiding API keys though: you need backend code to do this, which would require them writing Netlify serverless functions (requires quite a bit of Node knowledge).

Docker is a big gap atm. When I joined Ticketmaster I had to slowly pick up Docker here and there as I went because literally every project was using it. I would be in favour of covering the basics, especially since GitHub Actions is fully Docker-based (and I wanted to drop Travis in favour of that).

My main concern would be the database. DevOps for servers is one thing, but I've found it much harder for DBs. I had one failed attempt at getting Node and Postgres Docker containers to play nicely together a while ago so I'm not sure exactly how that would work. The nice thing about Heroku is they have a built-in Postgres setup that's super easy to use.

[oliverjam]

Also this part of GCP might put students off:

Make sure that billing is enabled for your Google Cloud project

[sofer]

This probably belongs in the mooted experimental Monday evening sessions https://github.com/foundersandcoders/london-programme/issues/1014#issuecomment-594075791

[oliverjam]

Yeah maybe. I did a test run deploying a Node project to Heroku using Docker and I think it wouldn't be too complicated. I hit a whole bunch of hard to debug issues, but if you follow the right instructions it should be fine.

So we could get them to deploy their Week 7 Node app with Docker.

[sofer]

I expect we could set up a FAC GCP account, which would get round the issue with billing (as long as don't forget to delete docker instances periodically).

[oliverjam]

The Heroku free tier seems to be the simplest way to get started, but I guess that might be useful if we want them to deploy more complex client stuff later on

[oliverjam]

Just discovered Node have a really comprehensive guide on getting started with Docker

https://nodejs.org/en/docs/guides/nodejs-docker-webapp/

[sofer]

Heroku has decent documentation, too https://devcenter.heroku.com/categories/deploying-with-docker.

It would be trivially easy to just point people at these resources and tell them to get on with it in pairs, either in Week 8 or as an additional research exercise (perhaps for the DevOps roles?) in Week 12.

[oliverjam]

I wouldn't say trivial. I have medium professional experience with Docker (and I already had it installed on my machine) and I still hit a few very frustrating and misleading errors.

❯ docker build .
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

❯ docker build .
ERRO[0000] failed to dial gRPC: unable to upgrade to h2c, received 502 
context canceled

I then spent about 30 minutes debugging why Heroku wouldn't work after I got a Docker image to actually build. In order:

1. I needed to expose a port in the image
2. I need to map the port in the image to be externally accessible
3. I had to convince Heroku to actually allocate resources to the deployed image
4. Something else I can't remember now went wrong too

[macintoshhelper]

Regarding using Docker with a database, my understanding is that using Docker for permanent data is quite dangerous and should generally be avoided, whether that be running a database, or storing flat files. It's best to either use a cloud database provider like AWS, DigitalOcean or GCP, or self-host a database on a VPS/bare metal server (or Heroku, which is a wrapper on top of AWS – with free use subsidised by higher commercial prices).

One idea could be to introduce students to some basic Linux sysadmin, and use a cheap VPS for deployment. Should be around £4 / month with OVH, or can be rented hourly from DigitalOcean/AWS/GCP, and the front-end can be cached as a static site on GitHub for archival/portfolio purposes when the server is torn down. Would be really valuable DevOps experience for students to set-up a firewall, SSH, database, Docker, etc on the machine. A VPS could be shared across multiple apps too (different database tables and screens or pm2 sessions). I'd be happy to provide/open-source the necessary bash scripts for this, and could do some Monday evening workshops on it.

Container registry could add complication (generally need to pay for monthly bandwidth and storage costs, or someone will be subsidising them). I think Docker's public registry should be fine, so long as environment variables/sensitive configs aren't built, since it's effectively an open-source registry. Docker and Kubernetes can be a bit of a time sink, so I would make sure that there are some ready-made resources on GitHub for the necessary configs, etc.

I think it's worth people using static sites where possible (say with Gatsby) and use GitHub Pages (for free hosting, substituted for AWS S3 and CloudFront in a professional setting), and CloudFlare as a free CDN (DDoS protection, bandwidth cost reduction, etc). Although a modular CDN/API/static site set-up might be overkill and not feasible in early weeks though.

[sofer]

I'd be happy to provide/open-source the necessary bash scripts for this, and could do some Monday evening workshops on it.

Yes, yes ❤️

@macintoshhelper I don't know when you are back in London, but I would love us to schedule a Monday session or two https://github.com/foundersandcoders/london-programme/issues/1014

[oliverjam]

I agree that we shouldn't be using Postgres inside Docker. I think the general idea was to stick with the current Node deployment approach (Heroku server, plus Heroku-hosted Postgres for DB week) for weeks 4, 5 & 6, then get them to Dockerize their server (but leave the DB the same) for week 7.

I think getting them to set up a VPS from scratch might be pushing "full stack" a bit too far. The course has limited time, and the students' brains have limited capacity for new concepts. I would rather keep the focus on web dev with just-enough-devops-to-get-by. It's also a skillset that is increasingly abstracted away (e.g. even at Ticketmaster with a full devops team they weren't configuring bare metal themselves).

I agree that this is a great topic for post-course meetups/sessions though.

[macintoshhelper]

I agree that we shouldn't be using Postgres inside Docker. I think the general idea was to stick with the current Node deployment approach (Heroku server, plus Heroku-hosted Postgres for DB week) for weeks 4, 5 & 6, then get them to Dockerize their server (but leave the DB the same) for week 7.

I think getting them to set up a VPS from scratch might be pushing "full stack" a bit too far. The course has limited time, and the students' brains have limited capacity for new concepts. I would rather keep the focus on web dev with just-enough-devops-to-get-by. It's also a skillset that is increasingly abstracted away (e.g. even at Ticketmaster with a full devops team they weren't configuring bare metal themselves).

I agree that this is a great topic for post-course meetups/sessions though.

Bare metal should be pretty straight forward, provided you have the necessary scripts, etc; I self-taught it before FAC. If you use Linux, you'll be familiar with most of it already. SSH set-up is the same as GitHub SSH keys (which everyone should really be using together with 2FA). DigitalOcean can abstract away most of the work with their one-click apps. People should learn to use a VPS before using AWS/GCP, etc imo, most of the concepts carry over like user permissioning (not using root/admin users) and firewalls; there's a lot of DevOps magic nowadays that can get people locked into paying a tonne of money down the line (e.g. $50+ / month for a basic Kubernetes setup), especially if you don't know how to use a CDN and end up paying lots for bandwidth.

One point though is that people should be using password managers (KeePass or LastPass are good) for any kind of DevOps.

Dokku seems pretty interesting, it's essentially a self-hosted drop-in replacement for Heroku that uses Docker and has plugins for Postgres, Let's Encrypt, etc, and has a one-click install with DigitalOcean, so can be spun up for a few hours or a week. I've used bare metal/VPSes for years, so know what the blocking points tend to be. My docs so far: https://github.com/coding-wiki/learn-devops/tree/master/server-management

TLDR; What I'd suggest is for front-end: standardising use of GitHub Pages (npm package gh-pages is great, for CLI deployment from a GitHub repo) for early weeks. And for back-end, say week 7 or 8, introduce Docker, or something like Dokku, depending on student's feedback.

[oliverjam]

We did a one-day project introducing Docker to FAC19 after the first 8 weeks, and I think we'll continue that going forward.