Maintenance mode for productions servers

[CarlosNZ]

Ideally we need some way to put the server into a "maintenance" mode, which just shows the user a helpful "unavailable" page (rather than an nginx fail page) while we take the server offline for updates (snapshot or build).

The simplest solution is just to temporarily re-direct everything to a simple HTML page that says "Server is currently offline for maintenance". However, there are other considerations:

• when downloading a snapshot to make offline changes, any data modified on the server before the snapshot is re-uploaded will be lost. So we need to make sure we make the system unavailable during this time.
• a simple re-direct works fine for new users coming to the site, however it doesn't work great for users already on the site who try and take actions. They won't be re-directed (unless they refresh the page) but their server queries will silently fail. Ideally we would have some kind of flag in the returned GraphQL that lets the front-end know the server is in "maintenance mode", and to re-direct to the Maintenance page. However, this is not simple to do, plus it relies on the server remaining active (which won't work when we actually stop and re-start the instance). Maybe we can just set the front-end to automatically re-direct whenever any query gets a 404 error from the server, but this might have other unwanted side-effects so needs some careful thinking about.

Proposed solution

We use nginx to re-direct according to certain rules, and we dynamically change and reload the nginx config file according to which "mode" we're in:

Modes

• Offline: All traffic is re-directed to maintenance page (on another server)
• Online: Normal functioning
• Admin:
  • All traffic is allowed to access the /login route
  • Nginx authenticates using JWT. If isAdmin token, pass traffic through as normal. But if anything else, re-direct to maintenance page

This means that Nginx and the Conforma instances need to share the same private key for JWT signing/validation. This should be straightforward -- we store the key in the host system somewhere, and pass it into Conforma via environment vars.

How to switch "Modes"

A shell script to update nginx config and reload. We could add the "offline" and "online" modes to the existing conforma launch scripts, and then run commands to change to "Admin" mode and back.

[CarlosNZ]

• Nginx authenticates using JWT. If isAdmin token, pass traffic through as normal. But if anything else, re-direct to maintenance page

Just realised that this is only available for nginx PLUS (paid) :(