Error while retrieving image from cache with 1.7

[jbub]

Actual behavior

Error while retrieving image from cache: golang:1.17-alpine strict validation requires the registry to be explicitly defined

Expected behavior

No error like with the 1.6 release.

To Reproduce Steps to reproduce the behavior:

1. Run executor with setup below.

Additional Information

• Dockerfile FROM golang:1.17-alpine AS build ...
• Build Context

  /kaniko/executor
  --dockerfile=path/to/Dockerfile
  --context=path/to/context
  --destination=my.registry.tld/my-image
  --registry-mirror=my.registry.tld
  --cache=true
  --cache-repo=my.registry.tld/cache
  --cache-ttl=3h
  --log-timestamp

• Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:v1.7.0

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [ ]
  Please check if this error is seen when you use --cache flag	- [x]
  Please check if your dockerfile is a multistage dockerfile	- [x]

[jbub]

Possibly related to 4e218f21623d89ef4161461c1f3c73baf2d7c119 ?

[darthf1]

I'm getting the following on 1.7, while everything worked with 1.6:

ERRO[0007] Error while retrieving image from cache: node:16.11.1-alpine repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./`: docker-registry:5000 

/kaniko/executor \
  --registry-mirror docker-registry:5000 \
  --insecure-pull \
  --cache \
  --context dir://client \
  --dockerfile ./client/Dockerfile \
  --destination ${CI_REGISTRY}/${CI_PROJECT_PATH}/client:${CI_COMMIT_SHORT_SHA}

Is this related or should I create a new ticket?

Edit: created https://github.com/GoogleContainerTools/kaniko/issues/1788

[jbub]

Probaly a different one but the cause may be the same.

[aaron-prindle]

@jbub I believe this was fixed in v1.8.0 - can you confirm if this issue was fixed for you or if you are still encountering this? Thanks

[juan-carvajal]

I am seeing the same behavior with latest image, cache has already been populated:

INFO[0000] Retrieving image manifest python:3.10-slim
INFO[0000] Retrieving image python:3.10-slim from registry index.docker.io
INFO[0000] Found sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94 in local cache
INFO[0000] Found manifest at /cache/sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94.json
INFO[0000] Image already in cache: python:3.10-slim
INFO[0000] Retrieving image manifest python:3.9-slim
INFO[0000] Retrieving image python:3.9-slim from registry index.docker.io
INFO[0001] Found sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3 in local cache
INFO[0001] Found manifest at /cache/sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3.json
INFO[0001] Image already in cache: python:3.9-slim
INFO[0001] Retrieving image manifest python:3.8-slim
INFO[0001] Retrieving image python:3.8-slim from registry index.docker.io
INFO[0002] Found sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b in local cache
INFO[0002] Found manifest at /cache/sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b.json
INFO[0002] Image already in cache: python:3.8-slim
INFO[0002] Retrieving image manifest python:3.7-slim
INFO[0002] Retrieving image python:3.7-slim from registry index.docker.io
INFO[0003] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache
INFO[0003] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json
INFO[0003] Image already in cache: python:3.7-slim
INFO[0003] Retrieving image manifest docker.io/library/python:3.10-slim
INFO[0003] Retrieving image docker.io/library/python:3.10-slim from registry index.docker.io
INFO[0004] Found sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94 in local cache
INFO[0004] Found manifest at /cache/sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94.json
INFO[0004] Image already in cache: docker.io/library/python:3.10-slim
INFO[0004] Retrieving image manifest docker.io/library/python:3.9-slim
INFO[0004] Retrieving image docker.io/library/python:3.9-slim from registry index.docker.io
INFO[0005] Found sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3 in local cache
INFO[0005] Found manifest at /cache/sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3.json
INFO[0005] Image already in cache: docker.io/library/python:3.9-slim
INFO[0005] Retrieving image manifest docker.io/library/python:3.8-slim
INFO[0005] Retrieving image docker.io/library/python:3.8-slim from registry index.docker.io
INFO[0006] Found sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b in local cache
INFO[0006] Found manifest at /cache/sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b.json
INFO[0006] Image already in cache: docker.io/library/python:3.8-slim
INFO[0006] Retrieving image manifest docker.io/library/python:3.7-slim
INFO[0006] Retrieving image docker.io/library/python:3.7-slim from registry index.docker.io
INFO[0006] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache
INFO[0006] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json
INFO[0006] Image already in cache: docker.io/library/python:3.7-slim
INFO[0006] Retrieving image manifest public.ecr.aws/docker/library/python:3.10-slim
INFO[0006] Retrieving image public.ecr.aws/docker/library/python:3.10-slim from registry public.ecr.aws
INFO[0007] Found sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94 in local cache
INFO[0007] Found manifest at /cache/sha256:8244f266195f442379acd0e7d8918b1ef3967050ebf49c40291e89d0a1ba8f94.json
INFO[0007] Image already in cache: public.ecr.aws/docker/library/python:3.10-slim
INFO[0007] Retrieving image manifest public.ecr.aws/docker/library/python:3.9-slim
INFO[0007] Retrieving image public.ecr.aws/docker/library/python:3.9-slim from registry public.ecr.aws
INFO[0008] Found sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3 in local cache
INFO[0008] Found manifest at /cache/sha256:4719115deb9cc7a5479a7d3c57cfceac2be89fcaf0fed8c747e8dfb4b01a79a3.json
INFO[0008] Image already in cache: public.ecr.aws/docker/library/python:3.9-slim
INFO[0008] Retrieving image manifest public.ecr.aws/docker/library/python:3.8-slim
INFO[0008] Retrieving image public.ecr.aws/docker/library/python:3.8-slim from registry public.ecr.aws
INFO[0009] Found sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b in local cache
INFO[0009] Found manifest at /cache/sha256:5af562618afca22211e56e80937c73d14923ad8fdef094f9084017fcdcdc6a8b.json
INFO[0009] Image already in cache: public.ecr.aws/docker/library/python:3.8-slim
INFO[0009] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim
INFO[0009] Retrieving image public.ecr.aws/docker/library/python:3.7-slim from registry public.ecr.aws
INFO[0010] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache
INFO[0010] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json
INFO[0010] Image already in cache: public.ecr.aws/docker/library/python:3.7-slim
time="2024-07-11T16:24:44.873Z" level=info msg="sub-process exited" argo=true error="<nil>"

But it seems that executor still needs to check the remote registry.

Successful run with cache:

INFO[0000] Resolved base name public.ecr.aws/docker/library/python:3.7-slim to compile-image 
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0000] Retrieving image public.ecr.aws/docker/library/python:3.7-slim from registry public.ecr.aws 
INFO[0001] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache 
INFO[0001] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json 
INFO[0001] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0001] Returning cached image manifest              
INFO[0001] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache 
INFO[0001] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json 
INFO[0001] Built cross stage deps: map[0:[/opt/venv /app]] 
INFO[0001] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0001] Returning cached image manifest              
INFO[0001] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache 
INFO[0001] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json 
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'public.ecr.aws/docker/library/python:3.7-slim' [idx: '0', base-idx: '-1'] 
INFO[0001] Checking for cached layer privateacr.azurecr.io/mlops-cache:1a374251ccf3272965b0ab09dd5f91b7201085f78e8fadd6d71f98ac1304ec5b... 
INFO[0001] Using caching version of cmd: RUN python -m venv /opt/venv 
INFO[0001] Checking for cached layer privateacr.azurecr.io/mlops-cache:8f3702841fbfe9570f99d91bc08a05cd4e8329c84e7a9ca5d5a8ee2720b009c1... 
INFO[0001] Using caching version of cmd: COPY requirements.txt . 
INFO[0001] Checking for cached layer privateacr.azurecr.io/mlops-cache:44e3f9faa0299fab2002f52c5e6f7c8ff095f13dd845724775f29954d2d380f4... 
INFO[0001] Using caching version of cmd: RUN pip install --upgrade pip     && pip install -r requirements.txt     && pip install seldon-core 
INFO[0001] Checking for cached layer privateacr.azurecr.io/mlops-cache:81cadd0be4c710d5afa220d9a1b6d6e810464cc541a7d68de6f65b0e95bacdd6... 
INFO[0001] No cached layer found for cmd COPY . .       
INFO[0001] Unpacking rootfs as cmd COPY . . requires it. 
INFO[0004] WORKDIR /app                                 
INFO[0004] Cmd: workdir                                 
INFO[0004] Changed working directory to /app            
INFO[0004] Creating directory /app with uid -1 and gid -1 
INFO[0004] Taking snapshot of files...                  
INFO[0004] RUN python -m venv /opt/venv                 
INFO[0004] Found cached layer, extracting to filesystem 
INFO[0005] ENV PATH="/opt/venv/bin:$PATH"               
INFO[0005] No files changed in this command, skipping snapshotting. 
INFO[0005] COPY requirements.txt .                      
INFO[0005] Found cached layer, extracting to filesystem 
INFO[0005] RUN pip install --upgrade pip     && pip install -r requirements.txt     && pip install seldon-core 
INFO[0005] Found cached layer, extracting to filesystem 
INFO[0012] COPY . .                                     
INFO[0012] Taking snapshot of files...                  
INFO[0012] Pushing layer privateacr.azurecr.io/mlops-cache:81cadd0be4c710d5afa220d9a1b6d6e810464cc541a7d68de6f65b0e95bacdd6 to cache now 
INFO[0012] Pushing image to privateacr.azurecr.io/mlops-cache:81cadd0be4c710d5afa220d9a1b6d6e810464cc541a7d68de6f65b0e95bacdd6 
INFO[0013] Pushed privateacr.azurecr.io/mlops-cache@sha256:acaa42d0b56061c7d14360ac7e5a4e13666a9a3dc71318c0263d822016e5849f 
INFO[0013] Saving file opt/venv for later use           
INFO[0015] Saving file app for later use                
INFO[0015] Deleting filesystem...                       
INFO[0016] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0016] Returning cached image manifest              
INFO[0016] Found sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be in local cache 
INFO[0016] Found manifest at /cache/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be.json 
INFO[0016] Executing 0 build triggers                   
INFO[0016] Building stage 'public.ecr.aws/docker/library/python:3.7-slim' [idx: '1', base-idx: '-1'] 
INFO[0017] Checking for cached layer privateacr.azurecr.io/mlops-cache:133c99ab64d5d9a52c79c19286e36b502503496aadeed60a9556a1a68e2543d9... 
INFO[0017] Using caching version of cmd: COPY --from=compile-image /opt/venv /opt/venv 
INFO[0017] Checking for cached layer privateacr.azurecr.io/mlops-cache:4b512421fcfab653e4d369fc8bf4d5f767c492c3b64f9a5227784b918a9302e8... 
INFO[0017] No cached layer found for cmd COPY --from=compile-image /app /app 
INFO[0017] Cmd: EXPOSE                                  
INFO[0017] Adding exposed port: 5000/tcp                
INFO[0017] Adding exposed port: 9000/tcp                
INFO[0017] Unpacking rootfs as cmd COPY --from=compile-image /app /app requires it. 
INFO[0020] WORKDIR /app                                 
INFO[0020] Cmd: workdir                                 
INFO[0020] Changed working directory to /app            
INFO[0020] Creating directory /app with uid -1 and gid -1 
INFO[0020] Taking snapshot of files...                  
INFO[0021] COPY --from=compile-image /opt/venv /opt/venv 
INFO[0021] Found cached layer, extracting to filesystem 
INFO[0028] COPY --from=compile-image /app /app          
INFO[0028] Taking snapshot of files...                  
INFO[0028] Pushing layer privateacr.azurecr.io/mlops-cache:4b512421fcfab653e4d369fc8bf4d5f767c492c3b64f9a5227784b918a9302e8 to cache now 
INFO[0028] ENV MODEL_NAME SKLearnServer                 
INFO[0028] Pushing image to privateacr.azurecr.io/mlops-cache:4b512421fcfab653e4d369fc8bf4d5f767c492c3b64f9a5227784b918a9302e8 
INFO[0028] No files changed in this command, skipping snapshotting. 
INFO[0028] ENV SERVICE_TYPE MODEL                       
INFO[0028] No files changed in this command, skipping snapshotting. 
INFO[0028] EXPOSE 5000 9000                             
INFO[0028] Cmd: EXPOSE                                  
INFO[0028] Adding exposed port: 5000/tcp                
INFO[0028] Adding exposed port: 9000/tcp                
INFO[0028] No files changed in this command, skipping snapshotting. 
INFO[0028] RUN chown -R 8888 /app                       
INFO[0028] Cmd: /bin/sh                                 
INFO[0028] Args: [-c chown -R 8888 /app]                
INFO[0028] Running: [/bin/sh -c chown -R 8888 /app]     
INFO[0028] Pushed privateacr.azurecr.io/mlops-cache@sha256:25d76c8d5092936310d79f68ad9656802a7940bb0051f94cd745578d0084a66f 
INFO[0029] Taking snapshot of files...                  
INFO[0029] ENV PATH="/opt/venv/bin:$PATH"               
INFO[0029] Pushing layer privateacr.azurecr.io/mlops-cache:37fb98ce93fd175a0ce0636c72e36ae83127570dbecf59d3f44f24b08b075134 to cache now 
INFO[0029] No files changed in this command, skipping snapshotting. 
INFO[0029] CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE 
INFO[0029] Pushing image to privateacr.azurecr.io/mlops-cache:37fb98ce93fd175a0ce0636c72e36ae83127570dbecf59d3f44f24b08b075134 
INFO[0029] No files changed in this command, skipping snapshotting. 
INFO[0029] Pushed privateacr.azurecr.io/mlops-cache@sha256:7b3b37b0b298e5ca33aa945ac8767f8dd675959cb78d23c0871f781dd5ad28f9 
INFO[0029] Pushing image to privateacr.azurecr.io/default-loadtestdhdepbouque:v1 
INFO[0031] Pushed privateacr.azurecr.io/default-loadtestdhdepbouque@sha256:31212c76df274adcf5f909c5a55ca0ebd3a258a637f20ba04e3c55112f918d6c 
time="2024-07-11T16:06:43.967Z" level=info msg="sub-process exited" argo=true error="<nil>"

Failed run with cache (rate limit):

INFO[0000] Resolved base name public.ecr.aws/docker/library/python:3.7-slim to compile-image 
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0000] Retrieving image public.ecr.aws/docker/library/python:3.7-slim from registry public.ecr.aws 
ERRO[0000] Error while retrieving image from cache: public.ecr.aws/docker/library/python:3.7-slim unable to complete operation after 0 attempts, last error: GET https://public.ecr.aws/v2/docker/library/python/manifests/3.7-slim: TOOMANYREQUESTS: Rate exceeded 
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.7-slim 
INFO[0000] Retrieving image public.ecr.aws/docker/library/python:3.7-slim from registry public.ecr.aws 
error building image: unable to complete operation after 0 attempts, last error: GET https://public.ecr.aws/v2/docker/library/python/manifests/sha256:ffd28e36ef37b3a4a24f6a771a48d7c5499ea42d6309ac911a3f699e122060be: TOOMANYREQUESTS: Rate exceeded
time="2024-07-11T16:19:42.055Z" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1
Stream closed EOF for ds-models-qa-weu-des-prod-testing-eu/default-loadtestdhdepbobiay5k5f5-build-and-push-3685358132 (main)

What is the point of having a cache layer that still needs to check remote CRs?