Failed ECR Push

[vitarkah]

Actual behavior error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "111111111.dk r.ecr.us-east-2.amazonaws.com/cs/aaa-svc:master": POST https://1111111111.dkr.ecr.us-east-2.amazonaws.com/v2/cs/aaa-svc/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized

To Reproduce Steps to reproduce the behavior: This portion is configured in argo workflow. So leveraging the relevant part from the workflow steps.

          - name: kaniko-build 
            inputs:
              parameters:
              - name: branch
              - name: repo                      
              - name: hash                      
            container:
              image: gcr.io/kaniko-project/executor:latest
              command:
                - /kaniko/executor                    
              args:
              - "--verbosity=debug"
              - "--dockerfile=Dockerfile"
              - "--context=git://xxxxxxxxxxx@github.com/eRecyclingCorps/xxxxxxx.git#refs/heads/master"
              - "--destination=11111111.dkr.ecr.us-east-2.amazonaws.com/cs/aaa-svc:master"
              env:
              - name: AWS_SDK_LOAD_CONFIG
                value: "true"
              - name: AWS_EC2_METADATA_DISABLED
                value: "true"
              - name: AWS_ECR_DISABLE_CACHE
                value: "true"      

Additional Information W DEBU[0000] Getting source context from git://xxxxxxxxxxxx@github.com/eRecyclingCorps/xxxxxxxxxx-engine.git#refs/heads/master DEBU[0000] Getting source from reference refs/heads/master Enumerating objects: 355, done. Counting objects: 100% (355/355), done. Compressing objects: 100% (265/265), done. Total 355 (delta 114), reused 251 (delta 35), pack-reused 0 DEBU[0000] Build context located at /kaniko/buildcontext/ DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile DEBU[0000] Cache disabled due to AWS_ECR_DISABLE_CACHE DEBU[0000] Retrieving credentials region=us-east-2 registry=11111111111 serverURL=1111111111.dkr.ecr.us-east-2.amazonaws.com service=ecr DEBU[0000] Calling ECR.GetAuthorizationToken registry=11111111111 error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "11111111111 .dk r.ecr.us-east-2.amazonaws.com/cs/aaa-svc:master": POST https://11111111111 .dkr.ecr.us-east-2.amazonaws.com/v2/cs/aaa-svc/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized Error: exit status 1

The node role has the policy - [EC2InstanceProfileForImageBuilderECRContainerBuilds]

I am unable to figure this out. Anything I try, get the same 401 error. 100% blocked!! Please help

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[cristian98149]

Hi, try to run again with --verbosity=trace. You should see some useful information.

[ajjamieson]

Hi,

I realize this is an older post but I've having the same issue and after a day of troubleshooting would love if anyone has any further troubleshooting steps or ideas.

DEBU[0000] Copying file /builds/#####/#####/#####/Dockerfile to /kaniko/Dockerfile 
TRAC[0000] Adding /var/run to default ignore list       
DEBU[0000] Cache disabled due to AWS_ECR_DISABLE_CACHE  
DEBU[0000] Retrieving credentials                        region=##### registry=#####serverURL=#####.dkr.ecr.#####.amazonaws.com service=ecr
DEBU[0000] Calling ECR.GetAuthorizationToken             registry=#####
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "#####.dkr.ecr.#####.amazonaws.com/#####:latest": POST https://#####.dkr.ecr.#####.amazonaws.com/v2/#####/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized

Thanks

PS: --verbosity=trace didnt give any more, just the same (trace output is above)

[kolja-lucht]

@ajjamieson we had the same issue and it took us a while to sort it out, too. We are in the process of setting up GitLab runners in AWS EKS and one CI/CD job in the GitLab pipeline is supposed to build and push an image to ECR. So if you're experiencing this issue in a Kubernetes context together with IRSA you might want to check if a) you're using the correct serviceAccount for the pod that's running the kaniko executor job and/or b) that the IAM role that is linked to your serviceAccount has the necessary permissions These are the permissions we're giving the IAM role:

{
  "Action": [
    "ecr:UploadLayerPart",
    "ecr:PutImage",
    "ecr:InitiateLayerUpload",
    "ecr:GetDownloadUrlForLayer",
    "ecr:CompleteLayerUpload",
    "ecr:BatchGetImage",
    "ecr:BatchCheckLayerAvailability"
  ],
  "Effect": "Allow",
  "Resource": "*",
},
{
  "Action": "ecr:GetAuthorizationToken",
  "Effect": "Allow",
  "Resource": "*",
}

Hope that helps 😃

[lvtujingji]

That is normaly when I add policy and changed yaml file .

The policy json { "Version": "2012-10-17", "Statement": [ { "Action": [ "ecr:UploadLayerPart", "ecr:PutImage", "ecr:InitiateLayerUpload", "ecr:GetDownloadUrlForLayer", "ecr:CompleteLayerUpload", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability" ], "Effect": "Allow", "Resource": "" }, { "Action": "ecr:GetAuthorizationToken", "Effect": "Allow", "Resource": "" } ] }

The yaml file apiVersion: v1 kind: Pod metadata: name: kaniko spec: containers:

• name: kaniko image: gcr.io/kaniko-project/executor:latest env:
  • name: AWS_SDK_LOAD_CONFIG value: "true"

    - name: AWS_EC2_METADATA_DISABLED

    value: "true"

    - name: AWS_ECR_DISABLE_CACHE

    value: "true"

    args:

  • "--verbosity=trace"
  • "--dockerfile=dockerfile"
  • "--context=git://github.com/lvtujingji/lvtujingji.git#refs/heads/main"
  • "--destination=account_id.dkr.ecr.us-east-2.amazonaws.com/adp-ecr-dev:nginx-v1.14" # replace with your dockerhub account volumeMounts:
  • name: ecrconfig mountPath: /kaniko/.docker/ restartPolicy: Never volumes:

• name: ecrconfig configMap: name: ecrconfig

  apiVersion: v1 kind: ConfigMap metadata: name: ecrconfig data: config.json: | { "credsStore": "ecr-login"}