buildkite-plugins / docker-buildkite-plugin

🐳📦 Run any build step in a Docker container
MIT License
113 stars 106 forks source link

Add "copy-checkout" config #182

Closed albertyw closed 2 years ago

albertyw commented 3 years ago

This adds a copy-checkout option that is similar to the mount-checkout option. Instead of mounting the working directory of test code, it copies the code into the container so that I can modify the test directory without polluting the host's copy.

I found that I needed this because I was running into one of a few scenarios when trying to install test dependencies:

  1. With default options, I would write dependencies into the container but it would break the next test because of root-owned files that the buildkite-agent wouldn't be able to delete.
  2. I tried running with propagate-uid-gid but then the user within the container would not have root permissions, needed for things like apt-get install.
  3. I could run with mount-checkout:false to not pollute the host's copy of the working directory, but I'd have to go through the kerfluffle of installing git, cloning the correct repository, checking out the correct branch, etc.
  4. I could also manually run post-test cleanup or immediately copy the working directory at the beginning of the test, but that seems hacky.

Therefore, I think the cleanest way for me is to use this copy-checkout config to get a copy of the code and have it isolated from the host's copy.

(I have yet to test out the docker-compose-buildkite-plugin, but maybe it also provides a workaround to my problem)

Fixes #113 Depends on #173 Replaces #177 which was an approach that doesn't work.

pda commented 3 years ago

I wonder if there's a simpler way to copy the workdir into the container before/during startup, rather than as a volume / command injection.

113 mentions using docker cp, but I gather that would need to be part of a multi-phase process of docker create …docker cp …docker start, instead of a simple docker run …. But maybe it's viable.

Or, do a docker build with Dockerfile from stdin, creating a temporary image to run instead of the upstream one:

# fake build vars
BUILDKITE_PLUGIN_DOCKER_IMAGE=alpine
BUILDKITE_JOB_ID=1234

# switcheroo
IMAGE_UPSTREAM="$BUILDKITE_PLUGIN_DOCKER_IMAGE"
BUILDKITE_PLUGIN_DOCKER_IMAGE="buildkite_image_$BUILDKITE_JOB_ID"

docker build --tag "$BUILDKITE_PLUGIN_DOCKER_IMAGE" -f- . <<EOF
FROM $IMAGE_UPSTREAM
COPY . /workdir
EOF

# Sending build context to Docker daemon    3.1kB
# Step 1/2 : FROM alpine
#  ---> a24bb4013296
# Step 2/2 : COPY . /workdir
#  ---> 05a616cb20b5
# Successfully built 05a616cb20b5
# Successfully tagged tmp_img_with_job_id:latest

docker run --rm "$BUILDKITE_PLUGIN_DOCKER_IMAGE" ls -al /workdir
# total 12
# drwxr-xr-x    2 root     root          4096 Mar  2 06:21 .
# drwxr-xr-x    1 root     root          4096 Mar  2 06:22 ..
# -rw-r--r--    1 root     root             6 Mar  2 06:13 hello
# -rw-r--r--    1 root     root             0 Mar  2 06:13 world
pzeballos commented 2 years ago

Hi @albertyw! Are you up on finishing the PR? Thanks!

pzeballos commented 2 years ago

If you are still interested in this functionality, you can create a new PR with the changes suggested, and we'll review it again 😊 Thanks!