Actual behavior
Trying to use kaniko with the registry mirror or registry-map flag, which includes a custom path for pulling images from DockerHub does not work. For example, I used an ECR registry setup with a pull through cache to DockerHub and passed as <account id>.dkr.ecr.us-west-2.amazonaws.com/docker-hub, where the custom path is docker-hub, and this was passed to the registry-mirror flag for pulling 2 DockerHub images referenced in the Dockerfile (one which existed in the default library namespace and one from a custom namespace), and both failed with the following errors:
WARN[0000] Failed to retrieve image docker-hub/alpine:3.19 from remapped registry <account id>.dkr.ecr.us-west-2.amazonaws.com: unable to complete operation after 0 attempts, last error: GET https://<account id>.dkr.ecr.us-west-2.amazonaws.com/v2/docker-hub/alpine/manifests/3.19: NAME_UNKNOWN: The repository with name 'docker-hub/alpine' does not exist in the registry with id '<account id>'. Will try with the next registry, or fallback to the original registry.
WARN[0000] Failed to retrieve image rancher/cowsay from remapped registry <account id>.dkr.ecr.us-west-2.amazonaws.com: unable to complete operation after 0 attempts, last error: GET https://<account id>.dkr.ecr.us-west-2.amazonaws.com/v2/rancher/cowsay/manifests/latest: NAME_UNKNOWN: The repository with name 'rancher/cowsay' does not exist in the registry with id '<account id>'. Will try with the next registry, or fallback to the original registry.
Expected behavior
Kaniko should be able to correctly use this mirror for pulling the images referenced in the Dockerfile, which point to DockerHub. For the alpine image it should use both the custom-path (docker-hub) and the default DockerHub namespace library when pulling the image. For the image which has its own custom namespace, it should be able to prefix just the custom path (docker-hub) when trying to pull the image.
To Reproduce
Steps to reproduce the behavior:
Create a registry mirror on ECR or any other registry of your choice, and setup that registry to have pull through cache to DockerHub, referenced within a path of the registry.
Pass the registry mirror to kaniko using the registry-mirror flag as --registry-mirror <registry mirror with a custom path> or the registry-map flag as --registry-map index.docker.io=<registry mirror with a custom path>. Make sure the --skip-default-registry-fallback is also used.
Use this mirror to build an image using a Dockerfile which uses an image from the default library namespace of DockerHub and one from a custom namespace.
Additional Information
Dockerfile
FROM alpine:3.19 AS build
RUN echo test > /test-file
FROM rancher/cowsay AS cowsay
RUN cowsay hello
FROM quay.io/containers/podman:v4.9.0 as podman
FROM alpine:3.19
RUN echo test
COPY --from=build /test-file /test-file
COPY --from=cowsay /usr/local/bin/cowsay /usr/local/bin/cowsay
COPY --from=podman /usr/bin/crun /usr/local/bin/crun
- Build Context
NA
- Kaniko Image (fully qualified with digest)
gcr.io/kaniko-project/executor:v1.22.0
**Triage Notes for the Maintainers**
<!-- ššš Thank you for an opening an issue !!! ššš
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->
| **Description** | **Yes/No** |
|----------------|---------------|
| Please check if this a new feature you are proposing | <ul><li>- [ ] </li></ul>|
| Please check if the build works in docker but not in kaniko | <ul><li>- [ ] </li></ul>|
| Please check if this error is seen when you use `--cache` flag | <ul><li>- [ ] </li></ul>|
| Please check if your dockerfile is a multistage dockerfile | <ul><li>- [ ] </li></ul>|
Actual behavior Trying to use
kaniko
with theregistry mirror
orregistry-map
flag, which includes a custom path for pulling images from DockerHub does not work. For example, I used an ECR registry setup with a pull through cache to DockerHub and passed as<account id>.dkr.ecr.us-west-2.amazonaws.com/docker-hub
, where the custom path isdocker-hub
, and this was passed to theregistry-mirror
flag for pulling 2 DockerHub images referenced in the Dockerfile (one which existed in the defaultlibrary
namespace and one from a custom namespace), and both failed with the following errors:WARN[0000] Failed to retrieve image docker-hub/alpine:3.19 from remapped registry <account id>.dkr.ecr.us-west-2.amazonaws.com: unable to complete operation after 0 attempts, last error: GET https://<account id>.dkr.ecr.us-west-2.amazonaws.com/v2/docker-hub/alpine/manifests/3.19: NAME_UNKNOWN: The repository with name 'docker-hub/alpine' does not exist in the registry with id '<account id>'. Will try with the next registry, or fallback to the original registry.
WARN[0000] Failed to retrieve image rancher/cowsay from remapped registry <account id>.dkr.ecr.us-west-2.amazonaws.com: unable to complete operation after 0 attempts, last error: GET https://<account id>.dkr.ecr.us-west-2.amazonaws.com/v2/rancher/cowsay/manifests/latest: NAME_UNKNOWN: The repository with name 'rancher/cowsay' does not exist in the registry with id '<account id>'. Will try with the next registry, or fallback to the original registry.
Expected behavior Kaniko should be able to correctly use this mirror for pulling the images referenced in the Dockerfile, which point to DockerHub. For the
alpine
image it should use both the custom-path (docker-hub
) and the default DockerHub namespacelibrary
when pulling the image. For the image which has its own custom namespace, it should be able to prefix just the custom path (docker-hub
) when trying to pull the image.To Reproduce Steps to reproduce the behavior:
kaniko
using theregistry-mirror
flag as--registry-mirror <registry mirror with a custom path>
or theregistry-map
flag as--registry-map index.docker.io=<registry mirror with a custom path>
. Make sure the--skip-default-registry-fallback
is also used.Dockerfile
which uses an image from the defaultlibrary
namespace of DockerHub and one from a custom namespace.Additional Information
FROM rancher/cowsay AS cowsay RUN cowsay hello
FROM quay.io/containers/podman:v4.9.0 as podman
FROM alpine:3.19 RUN echo test COPY --from=build /test-file /test-file COPY --from=cowsay /usr/local/bin/cowsay /usr/local/bin/cowsay COPY --from=podman /usr/bin/crun /usr/local/bin/crun