Simplify docker images down to three variants

[lox]

This extends on #38.

It simplifies down to three basic alpine based images, latest (stable), beta and experimental. Each contains the relevant agent and some basic stuff like su-exec, docker (client 1.13), jq and curl.

Images end up at around 230 MB.

[lox]

@toolmantim it looks like the edge image at https://download.buildkite.com/agent/experimental/3.0-beta.10.1323/buildkite-agent-linux-amd64 is returning a dynamically linked binary, so it's failing the test.

This at least verifies that it's worth having a test rather than auto-building via Docker Hub which I was seriously considering :)

[lox]

I updated to the latest experimental build and everything works fine.

[lox]

Using https://docs.docker.com/compose/install/#install-as-a-container for installing docker-compose.

[toolmantim]

Nice work! What was too hard about a local install of docker-compose?

This at least verifies that it's worth having a test rather than auto-building via Docker Hub which I was seriously considering :)

Something we can do is trigger the Docker Hub build from the Buildkite pipeline (they have a URL you can curl to kick it off), rather than on the GitHub change. If we wanted to do that.

[sj26]

This is looking nice! 👌

[lox]

So what do we have to do to move this forward?

[lox]

I've pushed images to https://hub.docker.com/r/lox24/buildkite-agent/ for this PR.

[lox]

Going to do some testing on a PR on https://github.com/buildkite/elastic-ci-stack-for-aws

[lox]

Btw @toolmantim, the hard bit about docker-compose is that it needs python installed. The current solution is a shim script that calls a docker-compose docker image, but I think that might end up being too complicated.

[toolmantim]

This seems almost ready for release!

The labels look the same, and I can't spot anything else that's a breaking change, aside from removing DIND.

I wonder what the best solution is for people wanting to add DIND support? Perhaps building your own custom image?

FROM buildkite/agent

ADD https://raw.githubusercontent.com/docker/docker/v.13.1/hack/dind /usr/local/bin
RUN chmod +x /usr/local/bin/dind

# copy and pasted from buildkite/agent base, with "dind" added
ENTRYPOINT ["dind", "/sbin/tini", "-g", "--", "ssh-env-config.sh", "/usr/local/bin/buildkite-agent"]

[toolmantim]

Related to @blueimp’s feedback in #38, perhaps instead of entrypoint.d we just wrap up our ENTRYPOINT commands in a single buildkite-agent-entrypoint executable so it can easily be called from images that build upon it?

So the above DIND example would become:

FROM buildkite/agent

ADD https://raw.githubusercontent.com/docker/docker/v.13.1/hack/dind /usr/local/bin
RUN chmod +x /usr/local/bin/dind

ENTRYPOINT ["dind", "buildkite-agent-entrypoint"]

Or does tini need to be at the beginning?

[lox]

IMO this is good to go. We might just need some docs updated.

[blueimp]

@toolmantim Yes, tini should be the first entrypoint binary so it can run as PID 1 and reap zombies.

[toolmantim]

Thanks @blueimp.

Okay, so I believe the updated DIND example would be this then:

FROM buildkite/agent

RUN curl -sL https://github.com/docker/docker/raw/v1.13.1/hack/dind > /docker-entrypoint.d/dind \
    && chmod +x /docker-entrypoint.d/dind

[toolmantim]

@rimusz do you know how this will affect https://github.com/buildkite/helm-charts? This removes DIND support from buildkite/agent. I noticed "Helm Classic" is the old world, so perhaps we need to deprecate the helm-charts anyhow?

[rimusz]

@toolmantim do not worry about the DIND removal, helm-charts repo will get new Helm v2 chart soon

[lox]

Updated docs @toolmantim

[toolmantim]

This looks great!

The last bit that I can see is that now is that we're versioning these images with the agent version, and we'll need to update the release process too so it's automated from the agent pipeline.

At the moment we just trigger the Docker pipeline and it rebuilds all images every time, no matter if it's a stable, beta or experimental release.

We can probably feed the agent version and SHA256 to the pipeline via the trigger, and have it automatically update the Readme & Dockerfiles via the GitHub API, which would trigger its own build?

So, perhaps 2 pipelines…

• pipeline.yml (tests and releases via GitHub events)
• pipeline.bump-version.yml (GitHub API update via trigger from agent release pipeline)

The agent pipeline does the GitHub API stuff for the homebrew recipe at the moment. Though it's a bit gross.

[toolmantim]

@rimusz thanks for clarifying!