Dockerize the setup

[amcewen]

As part of #16 (or maybe in prep for that), we should look at setting up the system on Docker. Probably want docker-compose with separate containers for:

• Postgres
• API server
• Website server
• Nginx as a front-end proxy

[ajlennon]

Sounds fun

[amcewen]

Assuming for now that we'll use docker-compose to set up three containers:

• web for the web server
• api for the API server
• db for a default postgres database server

The web and api servers will need their own Dockerfile to configure their container, we can use an off-the-shelf docker container for Postgres.

https://github.com/nodejs/docker-node/blob/main/README.md#how-to-use-this-image is a guide to using the official NodeJS container and extending it, and https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md seems a handy guide to read and apply for running them in production.

I've got the beginnings of a Dockerfile for the API server, but it's currently failing to install the dependencies, with node-sass seeming the most troublesome.

I think we might need to reorganise the directory structure of things, to split the API and web servers into separate projects with their own package.json and Dockerfile, but we'd need to work out how the dev and test environments would be affected by that. It can probably wait until I've got a working Dockerfile for at least the API server...

[amcewen]

This might have some useful tips - https://nickjanetakis.com/blog/best-practices-around-production-ready-web-apps-with-docker-compose

[amcewen]

Making some progress with this, all on the feature/17-dockerize-setup branch.

I'm splitting the package.json into two - one in api/ and one in website/, so that we can build them into separate docker containers.

In each folder there's also a Dockerfile which is responsible for building the container for that server. To build them you'd run (you might need to use sudo to run docker, I do on my Ubuntu machine):

cd api
docker build -t optimism/api .
cd ../website
docker build -t optimism/website .

Then running docker images should list two images: optimism/api and optimism/website. You need to then run the image to create the running container. There'll be a docker-compose config to manage that, but I've not done that yet. To test them individually I ran: docker run --name does-website -ti -p 3000:3000 --rm optimism/website or docker run --name does-api -ti -p 3001:3001 --rm optimism/api

If you're running one of the containers you can quit from it with Ctrl-P Ctrl-Q or run docker stop name-of-container in another terminal.

It won't do much (although the website container will serve you a perfectly good homepage) as the containers can't see each other, and the api container expects to connect to a Postgres database server, which will run in another container I've not set up yet. That plumbing will all come with the docker-compose setup

[amcewen]

I've added a docker-compose.yml config, which will build the docker images and then run up the three containers: optimism_db, optimism_api and optimism_website.

If you install docker-compose then you can run this to build and then start the containers:

docker-compose build
docker-compose up

There are an assortment of things still to do before this is ready:

• It should pull the database config from the environment, rather than having it hard-coded (#37 should resolve a lot of that, and we'll need to add the production environment)
• The environment variables in general need tidying up: the API URL is hard-coded in website/Dockerfile at present, and I'm not sure the other bits from .env get pulled in
• We need to work out how to run it for non-docker usage
• The package*.json files in root should be removed (unless some of the scripts to run the dev environment still need it)
• The postgres password needs to be changed, and moved into an environment variable
• We might want to move the code files in api/ and website/ into sub-folders so they're not sat at the same level as the package.json and Dockerfile? (not 100% sure this is necessary, but it feels like putting them in src/ would be neater?)
• Work out how we run migrations to set up the database (#37 will likely help here too)
• Work out how we populate the database

[ajlennon]

What you gonna run all this on? Balena? :)

[amcewen]

Hehe, nope. Digital Ocean is the current likely option.

[ajlennon]

Ooooh I can't stand them :)

[amcewen]

Well, you don't seem to be doing any of the work on it, so you're probably okay :stuck_out_tongue_closed_eyes:

With the docker-compose setup, we can run the database migration like this: docker-compose exec api npx knex migrate:latest --env production and then seed the database (we'll want a better solution for this long-term, but for now...) with docker-compose exec api npx knex seed:run --env production

[amcewen]

Looks like npm workspaces might be the way to set up the dev environment now that we have api/package.json and website/package.json. However, that pulls in a requirement for npm v7, and my machine is currently running npm v6, so I'll need to upgrade things first.

I had a quick look at the Digital Ocean droplets, and there's a docker version available as a standard build. If no-one complains before I'm next working on this (which will be at least after the 9th July) then I'll delete the old droplet we were using to experiment with Odoo, etc. and spin up a new docker one to host optimism.

[ajlennon]

True enough. My main issue with it is you do all this work and then find you can’t export your droplets without massive problems….

I wish somebody had told me that before I started using them :)

[amcewen]

Worked tonight on getting the development environment working with the new re-jigged structure for docker.

Tried adding two npm workspaces for the api and website sub-projects, and they're getting symlinked in node_modules as optimism-api and optimism (because those are the names I've set up in their package.json files).

Running npm run api, npm run dev-all, etc. from the root folder seems to work fine, and finds the database, etc. However, none of the knex files are working, because it either:

• Can't find the knexfile.js because that's moved into api/ now, or
• If I run it and specify the location of the knexfile: npx knex --debug --knexfile api/knexfile.js migrate:latest then it fails because it's trying to open ./db/optimism.sqlite3 when it's in the api folder.

Actually, I've just tried creating another knexfile.js in the root, which pulls in the real one from api/ and that seems to have done the trick. Not sure if the migrations are in the right place or not yet - probably not right now. So I need to check that more in both the running locally and in docker versions to ensure that it's set up right.

So far, despite setting up the workspaces, I'm not sure they're getting used for much beyond the recursive installation of modules when you run npm install - I don't seem to be using it to run the dev servers (for example) and haven't worked out if there's a way to run them as part of the workspace.

[amcewen]

Set up an initial version of the Droplet:

1. Created a droplet, on the basic level (we can upgrade if we need more memory, etc.), with the Docker on Ubuntu 20.04 from the Marketplace
2. Copy the IP address for the new droplet and set up the DNS for organisers.doesliverpool.com
3. ssh into the new droplet and make sure it's up to date with apt update and apt upgrade
4. Run ssh-keygen -t ed25519 -C "hello@doesliverpool.com" to create a new SSH key
5. Create a new deploy key (in https://github.com/DoESLiverpool/optimism/settings/keys/new) with the contents of ~/.ssh/id_ed25519.pub
6. Clone the repo to the new droplet

Then we'd be okay to build and run the docker setup, but I need to get that branch merged into develop and then master before that'll work.

[amcewen]

[https://gitlab.com/palisade/palisade-release/-/wikis/How-to-rebase-your-branch-from-the-master-branch was a handy reminder guide when I was rebasing my branch to get the latest changes to develop that @JackiePease had just merged]

I worked out how to get the migrations working properly, so I think this is all good to move to now.

I've merged the changed into develop - @JackiePease, you should pull the latest version before doing any more on it. You might need to run npm install from the root directory to update the dependencies now that things have been moved about a bit.

And I've pulled this latest develop over into master to then get it up and running on the production server. I've built and run the docker-compose setup, and http://organisers.doesliverpool.com:3000/ is working when the containers are running. Need to move that behind Nginx or similar, and make it autostart on boot. That'll be my next task.

[amcewen]

Worked on getting the basic server up and running...

1. Install nginx as a web server to sit in front of the site: apt install nginx
2. Copy the config file for the proxying: scp server-config/optimism-web organisers.doesliverpool.com:/etc/nginx/sites-available/
3. Enable the new config: ln -s /etc/nginx/sites-available/optimism-web /etc/nginx/sites-enabled
4. Disable the default server: rm /etc/nginx/sites-enabled/default
5. Restart nginx to pick up its new config: service nginx restart
6. Allow external access to port 80: ufw allow 80

It automatically restarts if the server gets rebooted, so I think this is now done. The remaining issues to get the server working well have been spun out as separate issues with the deployment tag