ECR login succeeds, but push fails with HEAD request 401 unauthorized

[aoyama-val]

Orb version

9.1.0

What happened

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@9.1.0
  aws-cli: circleci/aws-cli@4.1.3
workflows:
  build:
    jobs:
      - aws-ecr/build_and_push_image:
          context: *************
          repo: *************
          region: ${AWS_DEFAULT_REGION}
          tag: "${CIRCLE_SHA1}"
          auth:
            - aws-cli/setup:
              role_arn: ${AWS_IAM_ROLE_ARN}

#16 exporting to image
#16 exporting layers
#16 exporting layers 5.9s done
#16 exporting manifest sha256:6df314dcc346194eaf73963e15521ca9679586d95080a6efae886615b26306c0 0.0s done
#16 exporting config sha256:2069114fce5394cc9474b71dceb31fe1bbf781a27e349eeb61c8b7fa990f4a5e 0.0s done
#16 exporting attestation manifest sha256:d205a952b67cd4fd29396d25bbb4008368d2553bfbdb139a943572d1d7573b0d 0.0s done
#16 exporting manifest list sha256:d4b739fe2b3923dda7fe06893123cbfa061ef98eecf03d7dee037b7b414f281f 0.0s done
#16 pushing layers
#16 pushing layers 0.6s done
#16 ERROR: failed to push ************.dkr.ecr.**************.amazonaws.com/ticketing-rails:d4a654fb86a8e1c7ef4a912c49a96a73921aaea7: unexpected status from HEAD request to https://************.dkr.ecr.**************.amazonaws.com/v2/ticketing-rails/blobs/sha256:1c656b3fb894779c46614514306b4b427ed930d4f4bddd8d085b475bc49263d3: 401 Unauthorized
------
 > exporting to image:
------

 2 warnings found (use --debug to expand):
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 2)
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 12)
ERROR: failed to solve: failed to push ************.dkr.ecr.**************.amazonaws.com/*************:d4a654fb86a8e1c7ef4a912c49a96a73921aaea7: unexpected status from HEAD request to https://************.dkr.ecr.**************.amazonaws.com/v2/*************/blobs/sha256:1c656b3fb894779c46614514306b4b427ed930d4f4bddd8d085b475bc49263d3: 401 Unauthorized

Exited with code exit status 1

Expected behavior

Push succeeds.

[gonrial]

@aoyama-val can you check that the role stored in AWS_IAM_ROLE_ARN has permissions to push to that ECR repo? If you have permissions, but it still not working, can you list the permissions associated to that role?

[aoyama-val]

The role has these permissions at least:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ecr:BatchCheckLayerAvailability",
            "Resource": [
                "arn:aws:ecr:*****"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecr:CompleteLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:InitiateLayerUpload",
                "ecr:PutImage",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "arn:aws:ecr:*****"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

The config was working fine with aws-ecr@8.2.1 + aws-cli@3.1.4, however I faced the issue when tried to upgrade the orbs.

Thanks.

[aoyama-val]

I added Allow All permission to the role, but still push failed with the same error.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Also I changed the auth method from OIDC to access key/secret, and it worked fine. So I guess something about OIDC is wrong.

[gonrial]

@aoyama-val what is the definition for the role that you are using for OIDC? Pay special attention at the Condition section

[aoyama-val]

Carefully checked it but everything seemed correct. Also tried deleting the whole Condition, but nothing changed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.circleci.com/org/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
            },
            "Action": "sts:AssumeRoleWithWebIdentity"
        }
    ]
}