Speed up deployments (part 2)

[muloem]

Context

We have previously done some work to speed up deployments (https://github.com/akvo/akvo-flow/issues/2562) by using the GAE admin API. However, we had to revert it due the inability to set the correct scaling model for background instances. We have since phased out background instances therefore would like to reconsider the approach to fast deployments.

Problem or idea

We have to deploy to 100s of instances and this takes a long time and is scalable. We currently do push based deployments, i.e. code is deployed from a machine out to the instances. In the past we implemented pull based to speed up deployments and would like to look at this approach again.

Solution or next step

How could we solve it?

[iperdomo]

After some test and more thinking I think we're going back to deploy via gcloud app deploy and a new strategy for deploying when tagging.

Problems with GAE Admin API

While is really appealing to issue a POST request with a JSON payload for a new version, is not clear that with that request we can deploy changes to: Tasks, Indexes and Cron definitions. The current documentation of the GAE Admin API doesn't describe how to deploy changes to those other objects.

Inspecting gcloud's HTTP calls

gcloud has a flag to log all the HTTP requests issued during a process. After checking the result of this there are requests to other services like an internal Datastore API for configuring indexes, CloudTasks, etc.

After checking this it's clear that our interface to deployment must be gcloud app deploy and stop trying to fiddle with the internal details.

New strategy

• Merges to master get deployed to UAT2, new version os promoted directly
• New tags in master get deployed to UAT1 - new version is promoted directly. This allow us to test in UAT1 the candidate code.
• In parallel the same tag gets deployed to all instances in dark - this new version is not promoted, but available when needed
• After testing the tag in UAT1 we promote the tag, with a promote-* prefix, this goes into all instances and shift the traffic and the dark version becomes live

Deploy in parallel

• For deploying the tag to all instances in dark we're going to use gcloud app deploy.
• We could issue a Job in kubernetes that takes care of the deployment
• @dlebrero suggested to look at a different stage of the build in Semaphore, since our build are one-shot build+test+deploy. We want the tag to be available in UAT1 as soon as possible, and not be blocked by a deployment process to all instances.