Cache layers to filesystem (/cache folder) does't work

[DrMartiner]

Actual behavior kaniko could't cache layers to /cache

Expected behavior Kaniko will save layers to /cache and reuse the ones.

To Reproduce Steps to reproduce the behavior:

1. build base image on first step

   /kaniko/executor \
   --cache=true
   --cache-run-layers \
   --cache-copy-layers \
   --cache-dir=/cache \
   --cache-ttl=24 \
   --context /builds/project-name/tmp \
   --dockerfile /builds/project-name/tmp/.docker/Dockerfile
   --destination registry.gitlab.com/project-name/tmp:main_5d2b6401
   --insecure
   --skip-tls-verify

2. build image based on base image on second step

   /kaniko/executor \
   --cache=true
   --cache-run-layers \
   --cache-copy-layers \
   --cache-dir=/cache \
   --cache-ttl=24 \
   --context /builds/project-name/tmp \
   --dockerfile /builds/project-name/tmp/.docker/Dockerfile
   --destination registry.gitlab.com/project-name/tmp:main_5d2b6401
   --insecure
   --skip-tls-verify
   --cache-repo registry.gitlab.com/project-name/tmp:main_base
   --build-arg BASE_IMAGE=registry.gitlab.com/project-name/tmp:main_base

PS: I run GitLab runner in k8s

Additional Information

• .gitlab-ci.yml

  
  stages:
  - build_base
  - build

.build_kaniko: image: name: gcr.io/kaniko-project/executor:debug entrypoint: [ "" ] services:

• name: docker:20-dind alias: docker command: [ "--tls=false" ] variables: DOCKER_HOST: tcp://docker:2375/ DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "" DESTINATION: "" DOCKER_FILE_PATH: ".docker/Dockerfile" BUILD_ARGS: "" CACHE_ARGS: "" CACHE_DIR: /cache before_script:
• echo "{\"auths\":{\"${CI_REGISTRY}\":{\"username\":\"${CI_REGISTRY_USER}\",\"password\":\"${CI_REGISTRY_PASSWORD}\"}}}" > /kaniko/.docker/config.json script:
• /kaniko/executor --cache=true --cache-run-layers --cache-copy-layers --cache-dir=${CACHE_DIR} --cache-ttl=24 --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/${DOCKER_FILE_PATH}" --destination ${DESTINATION} --insecure --skip-tls-verify ${CACHE_ARGS} ${BUILD_ARGS}

build_base: stage: build_base extends: .build_kaniko variables: DESTINATION: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}_base DOCKER_FILE_PATH: ".docker/Dockerfile.base"

build: stage: build extends: .build_kaniko variables: DESTINATION: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REFNAME}${CI_COMMIT_SHORT_SHA} BUILD_ARGS: --build-arg BASE_IMAGE=${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}_base CACHE_ARGS: --cache-repo ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}_base

- .docker/Dockerfile
```Dockerfile
FROM ${BASE_IMAGE}

ADD . ${WORK_DIR}

• .docker/Dockerfile.base

  
  FROM python:3.9.0-slim

ENV WORK_DIR "/application/src"

RUN mkdir -p ${WORK_DIR} WORKDIR ${WORK_DIR}

RUN pip install --no-cache-dir poetry &&\ poetry config virtualenvs.create false ADD poetry.lock . ADD pyproject.toml . RUN poetry install

- helm-runner-values.yaml
```yml
gitlabUrl: https://gitlab.com
runnerRegistrationToken: token
concurrent: 55
checkInterval: 2
rbac:
  serviceAccountName: "gitlab-runner"
runners:
  image: "python:3.9.0-slim"
  executor: kubernetes
  privileged: true
  config: |
    [[runners]]
      name = "Kubernetes Runner"
      url = "https://gitlab.com/ci"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "default"
        image = "python:3.9.0-slim"
        privileged = true
        [[runners.kubernetes.volumes.pvc]]
          name = "docker-cache-tmp"
          mount_path = "/cache/"

• gitlab-runner-docker-cache-pvc.yaml

  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
  name: docker-cache-tmp
  spec:
  accessModes:
  - ReadWriteOnce
  resources:
  requests:
    storage: 4Gi

[tspearconquest]

Hi, fellow user here. Hopefully I can help.

It appears that in the first job, you're using kaniko with --cache-dir=/cache as though you're trying to populate the /cache directory with what's built.

The right way to populate the cache is by using kaniko warmer as documented here: https://github.com/GoogleContainerTools/kaniko#caching-base-images

Since you want to build your image and populate the cache with that image in one job, you'll probably want to play with kaniko warmer pulling down some dockerhub images to see how it populates the cache, and then replicate that process with your first build job, so that the PVC is properly setup with the files how kaniko expects them to be, as if kaniko warmer had been used to download it from your repo.

[aaron-prindle]

@DrMartiner were you able to get caching working here with the information in https://github.com/GoogleContainerTools/kaniko#caching-base-images and that @tspearconquest provided?

[aaron-prindle]

Closing this as I believe this is related to UX friction and not an issue with kaniko's caching directly. As @tspearconquest mentioned:

It appears that in the first job, you're using kaniko with --cache-dir=/cache as though you're trying to populate the /cache directory with what's built.

The right way to populate the cache is by using kaniko warmer as documented here: https://github.com/GoogleContainerTools/kaniko#caching-base-images