Dokku Deployment Setup - Githubissues

Updates and add config for deployment of Control using Dokku, as tested on our secondary hosting instance. Currently available at https://testing.staging.alces-flight.com

This allows for smooth deployment by setting a remote of dokku@secondary.apps.alces-flight.com:flight-control-staging and force pushing HEAD:master to that remote (appropriate ssh permissions required)
as defined in app.json, bin/predeploy.sh will run each time a deployment is made. This is based on the same file in Flight Hub, and will create the db if not present, and run any data migrations. This can be updated to add further actions in the future
Dokku will deploy separate containers for web (user interactions) and cron (scheduled tasks)
Adds new tasks, rake deployment:crontab:staging and rake deployment:crontab:production that will print out cron entries that need to be added to secondary or primary's crontab (NOT in the containers Dokku has created). This will need updating if scheduled tasks change, and means use of the whenever gem is now only applicable for development
It would be much better if we used a queuing system and had a specific container for processing background jobs (see #57). Dokku has a redis plugin, and we would need another Control resque container with a worker (see Flight Center's Procfile for an example)

Config changes

As each time a deployment is made, local files are (under normal circumstances) lost, as a new container is created and the old one destroyed, the credentials file is now included in the repo. As a result a suitable RAILS_MASTER_KEY is set as an environment variable for the dokku app
The credentials now only include a secret_key_base
This may interfere with running control on your local machine. I can either give you my master key, or you can delete the creds, generate your own and be careful not to include them in your branch
Slack token is now defined using an environment variable, as this is much easier to maintain with Dokku (otherwise a new slack token effectively needs to be hardcoded)
USD_GBP_CONVERSION, GBP_COMPUTE_CONVERSION and AT_RISK_CONVERSION are now also environment variables, so these are easier to update (as local changes to environment config files will be lost on redeployment). If not present, some hardcoded defaults are used
Database names are updated. You may therefore need to rename your local database (or manually edit your local database.yml and be careful not to add it to your branches)

Project config

The task rake projects:manage now adds an optional step when creating a project, for creating initial instance logs and generating a config file
if no project config file is found, a project will now use the defaut.yaml file instead of breaking
As these are local files, they would normally be lost after a dokku deployment. However, they have been retained by setting up some permanent storage (see https://dokku.com/docs~v0.22.9/advanced-usage/persistent-storage/#usage)
However, it would be better if this config were instead database records (see #68)

General Implications

This makes deployments very straightforward, with no manual intervention (usually) needed
In order to add and manage projects and users, admins will need access to secondary and primary. This is not great from a security/ safety perspective, so we should probably work to replace command line actions with some admin interface on the web application
Control is quite computation heavy and has a lot of repeated, timed tasks (so there are times where lots of calculations are being done in quick succession). These could also overlap times when Flight Center is running its cron tasks
Being on a shared machine means taking up too much of the available resources could impact other applications. We should therefore keep a careful eye on Control's usage (and secondary as a whole). During testing no problems seen with memory usage, but according to docker stats CPU usage can get quite high 90%+ for short periods
As mentioned above, having a background job system may help with this
We could/should also look at improving efficiency of forecasts. Calculating over budget switch offs is particularly complex, and likely has opportunities for efficiency improvements
We could look at some sort of caching system, so calculations aren't being repeated unnecessarily
If we need to scale the number of control instances it will not be feasible/ effective in this shared machine setup

openflighthpc / flight-control