Kaniko fails to 'warm' or build from existing cache due to docker hub's rate-limiting #3063

Open bossm8 opened 5 months ago

Actual behavior While running the the warmer in our GitLab CI/CD we quickly hit the rate-limiting with docker hub even though the cache is warmed already.

GitLab CI:

(...)

kaniko-pull:
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  stage: prepare
  script:
    - |
        for DF in $(find dockerfiles -type f); do
          /kaniko/warmer \
            --verbosity=info \
            --cache-dir=kaniko-cache \
            --dockerfile=${DF}
        done
  needs:
    - job: generate-dockerfiles
      artifacts: true
  cache:
    key: kaniko-cache
    untracked: true
    policy: pull-push
    paths:
      - kaniko-cache

kaniko-build:
  image: 
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  stage: build
  script:
    - |
        for DF in $(find dockerfiles -type f); do

            # Dockerfile format: Dockerfile_{{ .image_name }}_{{ .image_tag }}
            NAME=$(echo ${DF} | cut -d '_' -f 2)
            TAG=$(echo "${DF}" | cut -d '_' -f 3)

            echo -e "\nBuilding image tarball to ${NAME}_${TAG}.tar\n"

            /kaniko/executor \
                --cleanup \
                --no-push \
                --cache=true \
                --cache-repo=${IMAGE_BASE_NAME}/${NAME}-cache \
                --context=httpd \
                --dockerfile=${DF} \
                --tar-path=${NAME}_${TAG}.tar \
                --verbosity=info \
                --cache-dir=kaniko-cache \
                --destination=${IMAGE_BASE_NAME}/${NAME}:${TAG}
        done
  needs:
    - job: generate-dockerfiles
      artifacts: true
    - job: kaniko-pull
      artifacts: false
  artifacts:
    paths:
      - '*.tar'
    expire_in: 3h
  cache:
    key: kaniko-cache
    untracked: true
    paths:
      - kaniko-cache
    policy: pull

(...)

First run:

Checking cache for kaniko-cache-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Downloading artifacts
00:01
Downloading artifacts for generate-dockerfiles (1844[16](redacted)...
Downloading artifacts from coordinator... ok        host=redacted id=[18](https://redacted/-/jobs/184417#L18)4416 responseStatus=[20](redacted/-/jobs/184417#L20)0 OK token=glcbt-64
Executing "step_script" stage of the job script
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
$ for DF in $(find dockerfiles -type f); do # collapsed multi-line command
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0002] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0002] Returning cached image manifest              
INFO[0002] Found sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d3710[24](https://redacted/-/jobs/184208#L24)57f1 in local cache 
INFO[0002] Found manifest at kaniko-cache/sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0002] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0002] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0002] Returning cached image manifest              
INFO[0002] Found sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31 in local cache 
INFO[0002] Found manifest at kaniko-cache/sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31.json 
INFO[0002] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha[25](https://redacted/-/jobs/184208#L25)6:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Found sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0001] Found sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335 in local cache 
INFO[0001] Found manifest at kaniko-cache/sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335.json 
INFO[0001] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Found sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Found sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Found sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:45cc42828cc5ceeffa3a9b4f6363fb582fac3ab91f77bf403daa067f8f049f96.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:9 from registry registry.hub.docker.com 
INFO[0000] Found sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Returning cached image manifest              
INFO[0000] Found sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:ae32ae7f54215074bd0400dd82a4d543f8bc7d7d4b205563aa50f638dd20b335.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:9 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha256:69cecc7163282ad83e[27](https://redacted/-/jobs/184208#L27)b739fe8473b7c56e280e83827dcda60e5d37102457f1 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:69cecc7163[28](https://redacted/-/jobs/184208#L28)2ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee[31](https://redacted/-/jobs/184208#L31) in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:be879ad24fd5387ed135b99ebf0622c[32](https://redacted/-/jobs/184208#L32)3afab20ff7f1967d6f06e5dbf07ee31.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0000] Found sha256:be879ad24fd5387ed1[35](https://readcted/-/jobs/184208#L35)b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31 in local cache 
INFO[0000] Found manifest at kaniko-cache/sha256:be879ad24fd5387ed135b99ebf0622c323afab20ff7f1967d6f06e5dbf07ee31.json 
INFO[0000] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
Saving cache for successful job
00:10
Creating cache kaniko-cache-non_protected...
kaniko-cache: found 9 matching artifact files and directories 
untracked: found 16 files                          
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache

Second run (~10mins later):

Checking cache for kaniko-cache-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Downloading artifacts
00:01
Downloading artifacts for generate-dockerfiles (redacted)...
Downloading artifacts from coordinator... ok        host=redacted id=184416 responseStatus=200 OK token=glcbt-64
Executing "step_script" stage of the job script
00:03
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
$ for DF in $(find dockerfiles -type f); do # collapsed multi-line command
Caching base images for linux/amd64
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
INFO[0001] Found sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d3710[24](https://redacted/-/jobs/184417#L24)57f1 in local cache 
INFO[0001] Found manifest at kaniko-cache/sha[25](https://redacted/-/jobs/184417#L25)6:69cecc7163282ad83e[27](https://redacted/-/jobs/184417#L27)b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0001] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
INFO[0001] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0001] Returning cached image manifest              
INFO[0001] Found sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1 in local cache 
INFO[0001] Found manifest at kaniko-cache/sha256:69cecc7163282ad83e27b739fe8473b7c56e280e83827dcda60e5d37102457f1.json 
INFO[0001] Image already in cache: registry.hub.docker.com/library/rockylinux:8 
Caching base images for linux/arm64
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
WARN[0000] Error while trying to warm image: registry.hub.docker.com/library/rockylinux:8 Failed to retrieve image: registry.hub.docker.com/library/rockylinux:8: unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 
WARN[0000] Error while trying to warm image: registry.hub.docker.com/library/rockylinux:8 Failed to retrieve image: registry.hub.docker.com/library/rockylinux:8: unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
WARN[0001] Error while trying to warm image: registry.hub.docker.com/library/rockylinux:8 Failed to retrieve image: registry.hub.docker.com/library/rockylinux:8: unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 
WARN[0001] Error while trying to warm image: registry.hub.docker.com/library/rockylinux:8 Failed to retrieve image: registry.hub.docker.com/library/rockylinux:8: unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 
Failed warming cache: Failed to warm any of the given images
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Or the build:

Restoring cache
00:03
Checking cache for kaniko-cache-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Downloading artifacts
00:01
Downloading artifacts for generate-dockerfiles ([18](https://redacted/-/jobs/184471#L18)4443)...
Downloading artifacts from coordinator... ok        host=redacted id=184443 responseStatus=200 OK token=glcbt-64
Executing "step_script" stage of the job script
00:02
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca[19](https://redacted/-/jobs/184471#L19)e19e[20](https://redacted/-/jobs/184471#L20)223 ...
$ for DF in $(find dockerfiles -type f); do # collapsed multi-line command
Building image tarball to amd64_shib-php_1.0.0-php-8.0.tar
INFO[0000] Resolved base name registry.hub.docker.com/library/rockylinux:8 to shib-download 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
ERRO[0000] Error while retrieving image from cache: registry.hub.docker.com/library/rockylinux:8 unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 
INFO[0000] Retrieving image manifest registry.hub.docker.com/library/rockylinux:8 
INFO[0000] Retrieving image registry.hub.docker.com/library/rockylinux:8 from registry registry.hub.docker.com 
error building image: unable to complete operation after 0 attempts, last error: GET https://registry.hub.docker.com/v2/library/rockylinux/manifests/8: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Note: I also tried it with a different approach of running the warmer only once, which in the end tries to pull less images but else does not change anything:

KANIKO_ARGS=""
for DF in $(find dockerfiles -type f); do
  KANIKO_ARGS="--dockerfile ${DF} ${KANIKO_ARGS}"
done
/kaniko/warmer \
      --verbosity=${KANIKO_VERBOSITY} \
      --cache-dir=${KANIKO_CACHE_DIR} \
      --customPlatform=${PLATFORM} \
      ${KANIKO_ARGS}

Expected behavior Pipeline succeeds because manifests of base images are already present in the mounted cache. Note that there is one single runner VM, so the cache is present.

Not sure, but maybe this could be achieved by using the HEAD request instead of GET (https://docs.docker.com/docker-hub/download-rate-limit/)?

To Reproduce Steps to reproduce the behavior:

Unauthenticated docker hub user
Use dockerfiles with (many) base images

Additional Information

Dockerfile They are templated, which is why we loop them and add them all to the cache, but there is always only one single FROM entry in the files, in most cases even the same: FROM registry.hub.docker.com/library/rockylinux:9 and FROM registry.hub.docker.com/library/rockylinux:8
Build Context Only local files

Kaniko Image (fully qualified with digest) From the Pipeline:

Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use `--cache` flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [ ]

Very similar issue here: https://github.com/GoogleContainerTools/kaniko/issues/1787#issuecomment-2223377466

any update on this @bossm8 ?

@riyas-rawther unfortunately not, we switched back to dind as there were/are other bugs which affected us.

GoogleContainerTools / kaniko

Kaniko fails to 'warm' or build from existing cache due to docker hub's rate-limiting #3063