Use background jobs and resque

[timalces]

Aims to resolve #57. Dependent upon #69

Background Jobs Updates various rake tasks to no longer rely on fork to paralellise actions for multiple projects, instead converting them to background jobs.

This allows for a much more controlled completion of the tasks, especially on dokku where there otherwise may be a risk of control taking too much/ an unpredictable amount of host resources.

This is also important as most of these tasks are run multiple times a day and for every active project. It will also make scaling running these tasks (independent of the web app) more straightforward in the future.

These background tasks are sent to one of three queues, based on their priority: high, medium or low.

• recording instance logs: high
• checking for and running due change requests: high
• carrying out over budget switch offs: high
• utilisation monitor: medium
• recording cost logs: medium
• generating daily reports: low
• reporting on scheduled over budget switch offs: low
• recording instance sizes & prices: low
• syncing SSO accounts: low

Redis and Resque Adds implementation of Resque (which uses redis) for handling these background jobs.

• by default, it is assumed resque will be used in production
• by default, it is assumed background jobs will be performed 'inline' in development (performed immediately and sequentially)
• these can be overridden by assigning the QUEUE_ADAPTER environment variable
• if using resque, REDIS_URL must be set, or a default of localhost:6379 will be used

Note: if using resque (e.g. on staging) and you want to run a rake task with text output, you will need to override the queue adapter, as resque will print the text within its own process (and so not be visible in your console). E.g. QUEUE_ADAPTER=inline rake daily_reports:generate:all[true,true,true]

Resque Admin Interface Admins can view and take some actions on background jobs via the resque admin page, accessible at /resque. Note: this path will break if you have no redis service running

Dokku Config Adds config for using redis on dokku, with instructions for dokku to create a dedicated container to handle these background jobs

• this sets the priority order of the three queues (high, medium and low)
• this also generates 5 workers. This may require updating if using too much of secondary/primary's resources, or as the number of active projects grows
• As secondary/primary use an old version of redis (via the dokku-redis plugin), this creates a conflict in the REDIS_URL. This PR includes handling of altering the url to work both with a modern rails app and the old version of redis we are using. In the longer term we should look to update the dokku-redis plugin (not attempted here, as may impact flight center)
• a redis service has been set up and linked to this application on our secondary host
• Note: for some reason, after a deployment the 5 workers used by the earlier container remain in the resque worker list. A rake task, rake deployment:prune_resque_workers has been added, that should be run roughly 5 minutes after a deployment to remove these. However, it doesn't look like these dead workers remaining has any significant impact.

Testing note: if you are using redis on your local machine and you are also using Flight Center, you need to run separate redis services for each

[timalces]

Merging this as appears to be working. We should continue to monitor the workers, jobs and resource utilisation, especially when there are more projects, and therefore more background tasks being run.