server maintenance: firewall, docker services, regular checks

[bbest]

From @geocoug:

Ben,

Integral’s IT group (CC’d) would like to be more involved in the management of our cloud assets. Primarily, they would like to perform security monitoring and schedule regular maintenance windows for our Digital Ocean servers and volumes. The MarineEnergy.app server (206.189.173.168) and mounted backup volume are on that list.

For this to work smoothly, I think we should come up with some procedures/health checks that will be followed after server maintenance windows to verify our production applications are running smoothly. I’m pretty sure our entire environment is run with Docker (see docker-compose.yml), so it might be as straightforward as running a docker ps after reboot then verifying that https://marineenergy.app/ and https://shiny.marineenergy.app/report-v2/ do not return 404 errors. At the moment I don’t think our containers would start automatically after reboot but we may be able to update the restart flag so they do.

One of the things our IT group noted is that we currently don’t have firewall enabled. Can you confirm that these are the only ports that should be allowed so they can enable the firewall?

It’s been a while since we worked on this, so I’m probably forgetting some stuff.

Neil and Craig – please chime in if I’m missing anything.

Thanks, Caleb

[bbest]

The docker services are setup to restart unless stopped per:

https://github.com/marineenergy/server/blob/939315bfe88823b05eeb8668f848e5b88adfb480/docker-compose.yml#L60

Besides the ports for web services (80, 443) and secure shell / files (22), you can implement a firewall on the server to the outside world for all other ports, since those ports listed get used internally to the server for communication between containers. The only exception is the database port (5432), which you and I have in the past connected to directly from our personal machines. In future could still connect via SSH tunneling, a la this example.

[geocoug]

Thanks for jumping on this @bbest.

I'm not sure the unless-stopped option will work with daemon restarts. Might be something to test.

[bbest]

Hi @geocoug,

Ah, thanks for finding the nitty gritty details with this. Yes, let's update the restart policy to always. I made the changes in docker-compose.yml with this commit: restart: unless-stopped -> always #8 · marineenergy/server@8998207.

So I then I tried reloading the server, but it doesn't seem to take when I inspect:

# change directory to server repo
cd /home/bbest/mhkenvserversoftware

# show current policy
 docker inspect postgis | grep -A 3 RestartPolicy

            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },

# get latest changes from repo
git pull

# restart the service
docker-compose restart

Restarting postgis-backup ... done
Restarting rstudio        ... done
Restarting letsencrypt    ... done
Restarting proxy          ... done
Restarting nginx          ... done
Restarting nginx-dev      ... done
Restarting postgis        ... done

docker inspect postgis | grep -A 3 RestartPolicy

            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },

No dice. Ok, stop and start the containers.

docker-compose stop

Stopping postgis-backup ... done
Stopping rstudio        ... done
Stopping letsencrypt    ... done
Stopping proxy          ... done
Stopping nginx          ... done
Stopping nginx-dev      ... done
Stopping postgis        ... done

docker-compose start

Starting postgis        ... done
Starting postgis-backup ... done
Starting nginx          ... done
Starting proxy          ... done
Starting letsencrypt    ... done
Starting rstudio        ... done
Starting nginx-dev      ... done

docker inspect postgis | grep -A 3 RestartPolicy

            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },

Hmm... still not getting applied?

[geocoug]

Interesting quirk. Wonder if changes get applied when restarting the Docker daemon?

If you manually stop a container, its restart policy is ignored until the Docker daemon restarts or the container is manually restarted. This is another attempt to prevent a restart loop.

[bbest]

Good find! So I tried

sudo systemctl restart docker
docker inspect postgis | grep -A 3 RestartPolicy
# or
sudo systemctl stop  docker
sudo systemctl start  docker

Still no dice:

            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },

But then found this nugget in docker update | Docker Documentation:

Update a container’s restart policy (--restart)

You can change a container’s restart policy on a running container. The new restart policy takes effect instantly after you run docker update on a container.

To update restart policy for one or more containers:

docker update --restart=on-failure:3 abebf7571666 hopeful_morse

Note that if the container is started with “--rm” flag, you cannot update the restart policy for it. The > AutoRemove and RestartPolicy are mutually exclusive for the container.

docker update --restart=always postgis
docker inspect postgis | grep -A 3 RestartPolicy

And voila!

            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },

Run for other containers:

docker update --restart=always postgis-backup rstudio letsencrypt proxy nginx nginx-dev letsencrypt_server

[geocoug]

Phew. Think I'll give IT the go-ahead to start monitoring and point them to this issue for reference if things go sideways. Maybe we leave this issue open until they've worked through one of their maintenance routines. Objections?

Thanks @bbest!

[bbest]

Sounds good!