when using `--cache=true`, kaniko fail to push cache layer and (sometimes) fail to push image

[stavalfi]

Actual behavior

when using --cache=true, kaniko fail to push cache layer.

• in other repo, it failed to push the image it self because of the error.

error uploading layer to cache: failed to push to destination index.docker.io/stavalfi/kaniko-bug-reproduce/cache:64aaab8ce6f9a13b378fce81d74c2dd80d337417b4d0be39e0b879cf994c2e3d: HEAD https://index.docker.io/v2/stavalfi/kaniko-bug-reproduce/cache/blobs/sha256:79d3ac01f6735e269ed8804ee46cb0d5616826e6eaf4538f4e07f2b1c5c53835: unsupported status code 401

Expected behavior there shouldn't be any failures

To Reproduce Steps to reproduce the behavior:

I created a repo to reproduce the bug: https://github.com/stavalfi/kaniko-bug-reproduce the secret was created using the instruction of kaniko "getting started": https://github.com/GoogleContainerTools/kaniko/blob/master/docs/tutorial.md

Additional Information

• Dockerfile

  FROM ubuntu
  RUN echo "this line should not be printed in the log of the second build"
  ENTRYPOINT ["/bin/bash", "-c", "echo hello"]

• pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
    - name: kaniko
      image: gcr.io/kaniko-project/executor:debug-v1.2.0
      args: [
        '--cache=true',
        '--dockerfile=./dockerfile',
       '--context=git://github.com/stavalfi/kaniko-bug-reproduce.git#refs/heads/master',
        '--destination=stavalfi/kaniko-bug-reproduce:1.0.0'
        ]
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: dockerfile-claim

• Kaniko Image (fully qualified with digest)

gcr.io/kaniko-project/executor debug-v1.2.0 c91ffaa6289b (im not sure how to give more info)

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use --cache flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[stavalfi]

after investigation, a workaround is:

specifying --cache-repo that is different compared to --destination=stavalfi/kaniko-bug-reproduce:1.0.0 solved it:

'--cache-repo=stavalfi/kaniko-bug-cache',

I would expect from kaniko to have an error about it (if thats the desired behavior)

[bitsofinfo]

experiencing the same issue, seems random

[gaplo917]

experiencing the same issue. Added --cache=true will not able to push

INFO[0262] Pushing layer asia.gcr.io/xxxx/xxxx:3d99e9f12e13a0741b105f976920493168d520fc1b8d2a2566d4600b2410155f to cache now 

WARN[0263] error uploading layer to cache: failed to push to destination asia.gcr.io/xxxx/xxxx:yyyyyyy: GET https://asia.gcr.io/v2/token?scope=repository%xxxx%2Fxxxx%3Apush%2Cpull&service=asia.gcr.io: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Removing --cache=true is able to push

Seems using --cache=true --cache-repo=xxx is not work with GOOGLE_APPLICATION_CREDENTIALS

In GitLab CI,

variables:
  GOOGLE_APPLICATION_CREDENTIALS: $CI_PROJECT_DIR/google_auth.json
  IMAGE_URL: asia.gcr.io/xxx/$CI_PROJECT_NAME

publish:
  tags:
    - microk8s
  image: gcr.io/kaniko-project/executor:debug-v1.3.0
  stage: publish
  script:
    - cp $gcr_key $CI_PROJECT_DIR/google_auth.json
    - >
      /kaniko/executor
      --cache=true 
      --cache-repo=$IMAGE_URL
      --log-timestamp=true
      --snapshotMode=redo
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $IMAGE_URL:$TAG
      --build-arg=DEPLOY_VERSION=${TAG}

[michael-burt]

I am experiencing something similar. When I specify both --no-push and also --cache-repo in a single Kaniko build step, the Kaniko executor is unable to push to the cache registry. In the same cloubuild.yaml configuration file, subsequent build steps are able to push the their respective registries.

My use case is building a base image which shares a common cache repository with downstream stages to reduce build time.