Builds are generated from a set of versions

[lox]

Currently it's difficult to keep track of which files produce which versions, so this approach uses the new Docker 1.9 build arguments to parametize a smaller set of generic Dockerfile's with the versions of the things that go in them. The result is a standardized (and fairly optimized) hierarchy of images and tags.

The structure for the non-docker images looks like {buildkite_version}-{distro}, e.g edge-ubuntu. The stable builds omit the stable, so are simply ubuntu or alpine (the distro).

The docker images derive from the above base images, but add a version of docker. The idea would be to keep to only support the most recent point release of a minor version series. The tags created for these are {buildkite_version}-{distro}-docker-{docker_version}, e.g edge-ubuntu-docker-1.6.2. These are tagged with the docker major version too, so edge-ubuntu-docker-1.6 works.

Todo:

• [x] Check whether docker-ssh-env-config can be applied
• [x] Get dind working conditionally, perhaps via an ENV?
• [x] Get tests working again
• [x] Document shell scripts more
• [x] Update changelog / readme
• [x] Figure out what to do about ECS image

[toolmantim]

This is awesome. I say we do this.

Check whether docker-ssh-env-config can be applied

What's the question here?

Use bk pipeline to do builds in parallel

I wouldn't worry about this for now perhaps? Just keep it simples. Unless we end up doing something like that for tests anyhoo.

Figure out what to do about ECS image

I think we just ditch the ECS image and I replace it with docs when it comes time to ship the ECS stuff. That way we can simply update the docs when ECS updates their Docker versions etc, and not mess around with software. People can update their services/tasks/actions as they need based on the latest docs.

Anything I can help with?

[lox]

@toolmantim it looked like some images had older versions of docker-ssh-env-config applied and some not at all. I just wanted to be sure that there wasn't some good reason to not have it standard.

[lox]

@toolmantim are you ok with having to run a docker 1.9 CI worker to do these builds?

[lox]

@toolmantim how do you want to approach backwards compatibility? Documentation, or shall we add tags to support the current tag schema?

[lox]

I'm torn on whether to have the docker images conditionally start a dind server, perhaps based on a DIND env var, or whether to also tag a -dind variant. Thoughts?

[toolmantim]

are you ok with having to run a docker 1.9 CI worker to do these builds?

yep!

how do you want to approach backwards compatibility? Documentation, or shall we add tags to support the current tag schema?

The tags will just stay there as-is (only way to delete them is to email Docker Hub support anywho), and we can update the Readme for the new tags (even include a "deprecated tags" list if we wanted)?

I'm torn on whether to have the docker images conditionally start a dind server, perhaps based on a DIND env var, or whether to also tag a -dind variant. Thoughts?

Hmm that's an interesting idea, using an environment variable! I think that's not a bad idea, just to reduce the number of images and support footprint. Have it off my default.

[lox]

Ok cool, I think we are good to go then. Merge at your convenience @toolmantim.