CodeBuild backend

[ejholmes]

This is some polish on top of https://github.com/remind101/conveyor/pull/67.

Mainly:

• Tests
• Better handling of cancellations (build timeouts, shutdown signals, etc).
• Handling of failed builds
• Handling of branchless builds
• Handling of dry run flag.
• Handling of [docker nocache] flag.
• Ensures that CodeBuild project names can't collide.

I've tested this quite a bit locally, and it works pretty nicely (albeit, a little slower than the native Docker backend).

[Lowercases]

I didn't see any obvious problems :+1: I'll take another look in the morning with a fresh head :)

[ejholmes]

One thing that sucks about this is I keep hitting cloudwatch logs throttling, because of the io.Copy from the CodeBuild log stream, over to the Conveyor specific log stream. I'll follow up tomorrow by just making Conveyor read directly from the CodeBuild log streams, to reduce a bunch of unnecessary I/O.

[ejholmes]

Also, seems like docker build isn't using the existing layer cache, regardless of docker pull'ing before. Gotta look into that.

[ejholmes]

Alright, turns out that Docker 1.10 -> 1.12 doesn't use cached layers from docker pull, like Docker 1.8/1.9 did: https://github.com/moby/moby/issues/20316

At the moment, the official codebuild images that include a working Docker-in-Docker only include Docker 1.11.2 and Docker 1.12.1, which means, we can't use these images and get the existing layer caching behavior. So there's three possible ways forward:

1. Build a docker 1.8/1.9 CodeBuild compatible Docker-in-Docker image, so we can use the existing caching logic.
2. Build a Docker 1.13+ CodeBuild compatible Docker-in-Docker image and use --cache-from.
3. Use the current official CodeBuild images, and put/restore layer caches with S3 (gross).

I really want to avoid 3, so I'll investigate doing 1/2.

[ejholmes]

I think 1/2 might already be ruled out...turns out the official aws/codebuild/docker* images can't be run without --privileged, and there's no way to pass that down to CodeBuild.

I'm guessing that their official images get the --privileged flag since they're owned and managed by AWS. I'll open a ticket with them to verify.

[phobologic]

Ugh, this sucks. Way to pick the couple of versions we can't use, AWS.

[ejholmes]

Confirmed with AWS support that it's not possible to build a custom image that runs DiD on top of CodeBuild. Gonna put this on hold until there's an official 1.13 image.

[ejholmes]

Alright, I revived this. Thanks to some updates in CodeBuild, you can now enable privileged mode for custom docker containers. I have a branch of remind101/conveyor-builder that provides a Docker 1.8.1 image that's compatible with CodeBuild.

With these changes, layer caching works like it did before and unblocks this 🎉

[ejholmes]

Gonna close this PR for now. This ended up not working all that great. Alternatively, now that AWS Fargate is out, that's actually a potentially better fit than CodeBuild for Conveyor.