Multi stage Docker build cache documentation improvement

[csidyel]

Feedback from a customer:

I came across https://docs.semaphoreci.com/article/81-docker-layer-caching#about---cache-from. My Dockerfile uses multiple FROM, a.k.a. multi stage build image. By default --cache-from will not fully work with that. I manage to fix/workaround it, I have an example attached that could be used as an improvement to the documentation.

He attached the following file:

# For more Ruby information and examples, see
# https://docs.semaphoreci.com/article/73-ruby
version: v1.0
name: Example Ruby API
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    task:
      secrets:
        - name: quay-pull-secrets
      prologue:
        commands:
          - checkout
      jobs:
        - name: Build Docker image
          # With workaround for multi-stage cache: https://github.com/moby/moby/issues/34715
          commands:
            - echo ${DOCKER_PASSWORD} | docker login ${DOCKER_URL} --username ${DOCKER_USERNAME} --password-stdin
            - image=quay.io/mycompany/example-ruby-api
            - commit_hash=$(git log -n 1 --pretty=format:'%H')
            - docker pull $(grep -oE 'FROM .+$' Dockerfile | head -n 1 | cut -d ' ' -f 2) || true
            - docker pull ${image}:${SEMAPHORE_GIT_BRANCH}_builder || true
            - docker pull ${image}:${SEMAPHORE_GIT_BRANCH} || true
            - docker build --cache-from ${image}:${SEMAPHORE_GIT_BRANCH}_builder -t ${image}:${SEMAPHORE_GIT_BRANCH}_builder --target builder .
            - docker build --cache-from ${image}:${SEMAPHORE_GIT_BRANCH},${image}:${SEMAPHORE_GIT_BRANCH}_builder -t ${image}:${SEMAPHORE_GIT_BRANCH} .
            - docker push ${image}:${SEMAPHORE_GIT_BRANCH}_builder
            - docker push ${image}:${SEMAPHORE_GIT_BRANCH}
            - docker tag ${image}:${SEMAPHORE_GIT_BRANCH} ${image}:${commit_hash} && docker push ${image}:${commit_hash}
            - if [ -n "$(git tag --points-at HEAD)" ]; then git_tag=$(git describe --tags) && docker tag ${image}:${SEMAPHORE_GIT_BRANCH} ${image}:${git_tag} && docker push ${image}:${git_tag}; fi
            - if [ "$(git rev-parse --abbrev-ref HEAD)" == "develop" ]; then docker tag ${image}:${SEMAPHORE_GIT_BRANCH} ${image}:latest && docker push ${image}:latest; fi

[csidyel]

The customer attached an updated file:

# For more Ruby information and examples, see
# https://docs.semaphoreci.com/article/73-ruby
version: v1.0
name: Example Ruby API
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    task:
      secrets:
        - name: quay-pull-secrets
      prologue:
        commands:
          - checkout
      jobs:
        - name: Build Docker image
          # With workaround for multi-stage cache: https://github.com/moby/moby/issues/34715
          commands:
            - echo ${DOCKER_PASSWORD} | docker login ${DOCKER_URL} --username ${DOCKER_USERNAME} --password-stdin
            - image=quay.io/mycompany/example-ruby-api
            - image_tag=${SEMAPHORE_GIT_BRANCH/refs\/tags\//}
            - image_tag=${image_tag/\//-}
            - commit_hash=$(git log -n 1 --pretty=format:'%H')
            - docker pull $(grep -oE 'FROM .+$' Dockerfile | head -n 1 | cut -d ' ' -f 2) || true
            - docker pull ${image}:${image_tag}_builder || true
            - docker pull ${image}:${image_tag} || true
            - docker build --cache-from ${image}:${image_tag}_builder -t ${image}:${image_tag}_builder --target builder .
            - docker build --cache-from ${image}:${image_tag},${image}:${image_tag}_builder -t ${image}:${image_tag} .
            - if [[ "$SEMAPHORE_GIT_BRANCH" != *"refs/tags/"* ]]; then docker push ${image}:${image_tag}_builder; fi
            - docker push ${image}:${image_tag}
            - if [[ "$SEMAPHORE_GIT_BRANCH" != *"refs/tags/"* ]]; then docker tag ${image}:${image_tag} ${image}:${commit_hash} && docker push ${image}:${commit_hash}; fi
            - if [ "$(git rev-parse --abbrev-ref HEAD)" == "develop" ]; then docker tag ${image}:${image_tag} ${image}:latest && docker push ${image}:latest; fi

[AlyxPractice]

Hello, I'm very interested in this too. I tried many things, but could not make it work. I have 7 layers (4 for the release, 2 for local developers tools and 1 for testing purposes, local and CI/CD) so maybe that's why it's a bit more complicated to make it work.

Thanks for working on this 🙏

[csidyel]

@CommanderK5 Can you take a look at this? Whenever you have the time 😄