853: Service availability in AWS

[chriswilty]

Description

It came to my attention that the fargate service and associated infrastructure is costing us more than $3 a day. We can decrease that by around 70% if we bring the service down overnight, and not have the application up over weekends.

Currently there is no trivial way to achieve this using CDK, although there are some alpha constructs in development which we can use, together with a lambda function, to update the service to flip "desired task count" from zero to 1 in the morning and back to zero in the evening.

Resolves #853

Screenshots

Concerns

• We might want to show a different UI page outside of uptime hours, such as a generic "downtime / maintenance" message. I will assess how this looks once it's live in AWS, and spawn another issue if needed.
• I'm happy using the alpha constructs, as they seem robust; it's just that the API is subject to change so we may need to make minor adjustments whenever bumping to a new version.

Checklist

Have you done the following?

• [x] Linked the relevant Issue
• [ ] Added tests
• [ ] Ensured the workflow steps are passing

[chriswilty]

@pmarsh-scottlogic Understood! It's more complex than I would've liked, but the idea is that we schedule a pair of events, one in morning and one in evening, and when they run, the scheduler turns them into an "action" to run a lambda function. That lambda function has one job to do: adjust the number of tasks/containers the service is running: to 1 in the morning, 0 in the evening.

But we need to adjust the service definition itself, because simply auto-scaling the running service to zero will cause Fargate to check its task definition and see that it needs at least one container running, so it immediately brings one up again. This is Fargate's job - it manages container deployment for us, optimising according to current traffic and our defined limits. But our "desired count" is the baseline to which our number of containers will return under "normal" load, and that desired count cannot be zero, as that would make no sense. Hence why this is a touch more complex than I would've liked, and hence the need for a lambda function to adjust the service definition.