GCR auth broken when --no-push is set since v0.20.0

[firaxis]

Actual behavior When running the docker kaniko with --no-push, cache is not available and pushing new cache layers fails.

Expected behavior Docker image should be built using cache.

To Reproduce

/usr/bin/docker run --rm -v ~/.config/gcloud/:/root/.config/gcloud -v ~/workspace:/workspace 
gcr.io/kaniko-project/executor:v0.22.0 
--destination us.gcr.io/<your project>/kaniko/test 
--context dir:///workspace/ 
--dockerfile Dockerfile 
--cache=true 
--no-push 
--cache-repo us.gcr.io/<your project>/kaniko/test/cache

Additional Information

• Dockerfile

  FROM debian:9
  RUN apt update
  RUN apt -y install vim-tiny

  Triage Notes for the Maintainers The latest version it works is v0.19.0

  /usr/bin/docker run --rm -v ~/.config/gcloud/:/root/.config/gcloud -v ~/workspace:/workspace 
  gcr.io/kaniko-project/executor:v0.19.0
  --destination us.gcr.io/<your project>/kaniko/test 
  --context dir:///workspace/ 
  --dockerfile Dockerfile 
  --cache=true 
  --no-push 
  --cache-repo us.gcr.io/<your project>/kaniko/test/cache

  Description	Yes/No
  Please check if this a new feature you are proposing	NO
  Please check if the build works in docker but not in kaniko	NO
  Please check if this error is seen when you use --cache flag	NO
  Please check if your dockerfile is a multistage dockerfile	NO

[sfynx]

This also breaks any FROM from a GCR repository that depends on authentication.

[sameersbn]

i have the same issue. I worked around it by removing --no-push and pushing the image with the tag _ so that the cache is updated

[ejose19]

Neither reading the existing cache nor pushing to it works, as you mentioned last working version is 0.19.0

Msg when reading cache

INFO[0002] No cached layer found for ...

Msg when trying to push to cache

INFO[0222] Pushing layer .... to cache now
WARN[0222] error uploading layer to cache: failed to push to destination ... UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

This is critical since without cache builds take a long time, and removing --no-push is not an option since one may just want the tar image to use in next steps of cloudbuild only.

[virth]

We have the same issue with a FROM for an image in GCR (submitted with #1347).

[nesl247]

Running into this when trying to test out kaniko locally. We shouldn't need to push to be able to test out local builds.

[ejose19]

I've digged into the diffs from v0.19.0 to 0.20.0 and found the issue is because the credential-helper is no longer being called by default

https://github.com/GoogleContainerTools/kaniko/commit/8a020010b75852d5180f20f768fd09458175fd46#diff-2a33e645dffdb8c6e5dc53153fc455a6

The only other place where the credential-helper may be called is in this function

https://github.com/GoogleContainerTools/kaniko/blob/6c70f167dffde75618dc36c016414c8bcbc797e5/pkg/executor/push.go#L104-L154

However as we can see, there're are 2 issues with it

• If no push is set, it doesn't even run so credentials never get configured
• If we're using a destination that's not in gcr domain (since we're using --noPush, that's what it will be most likely) then even if the noPush "if" were removed we would get an error like POST https://index.docker.io/v2/library/$YOUR_DESTINATION_TAG UNAUTHORIZED: authentication required;

So either we add another block for credential-helper to run when the --noPush is set and gcr is set as cache, or CheckPushPermissions needs a refactor to support this use case.

While a PR comes, here's the workaround:

- name: 'gcr.io/kaniko-project/executor:debug-v1.2.0'
    id: my_step
    entrypoint: sh
    args:
      - '-c'
      - |
        docker-credential-gcr configure-docker -registries us.gcr.io &&
        executor \
        --dockerfile=Dockerfile \
        --cache=true \
        --cache-repo=us.gcr.io/$PROJECT/$NAME \
        --no-push \
        --destination=my-image:latest \
        --tarPath=/workspace/my-image.tar

Just call docker-credential-gcr configure-docker -registries us.gcr.io before executor (adapt the registry name if you're not using the US registry) and it will both retrieve cached layers and push cache layers without issues.

[tejal29]

Reopening as it was broken in a recent commit.