Eventual consistency error with ECR `GetRepositoryPolicy` / `SetRepositoryPolicy` (?)

[blakepettersson]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.8 on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.29.0

Affected Resource(s)

• aws_ecr_repository_policy
• aws_iam_role_policy

Terraform Configuration Files

data "aws_caller_identity" "current" {}

locals {
  atlantis_role_name = "custom-atlantis-ecr"
  account_id           = data.aws_caller_identity.current.account_id
  replication_regions  = ["eu-central-1", "us-east-1", "us-east-2", "ap-southeast-2", "eu-west-1"]
  repositories = {
    "11" = ["arn:aws:iam::000000000000:root"]
    "22" = ["arn:aws:iam::000000000000:root"]
  }
  pull_image_permissions = {
    Sid    = "PullImages",
    Effect = "Allow",
    Action = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:GetRepositoryPolicy",
      "ecr:DescribeRepositories",
      "ecr:ListImages",
      "ecr:DescribeImages",
      "ecr:BatchGetImage",
      "ecr:DescribeImageScanFindings"
    ],
    Principal = { AWS = "*" }
  }
}

resource "aws_iam_role" "atlantis-role" {
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : ["*"]
        },
        "Action" : "sts:AssumeRole"
      }
    ]
  })

  name                = local.atlantis_role_name
  managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonVPCReadOnlyAccess"]
}

resource "aws_iam_role_policy" "create-repository-policy" {
  name = "${local.atlantis_role_name}-policy"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action   = ["ecr:CreateRepository", "ecr:DescribeRegistry", "ecr:GetRegistryScanningConfiguration", "ecr:DescribeRepositories", "ecr:ListTagsForResource"]
          Effect   = "Allow"
          Resource = "*"
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_iam_role_policy" "ecr-policy" {
  name     = "${local.atlantis_role_name}-${each.key}-policy"
  for_each = local.repositories
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action   = ["ecr:DeleteRepository", "ecr:DeleteRepositoryPolicy", "ecr:SetRepositoryPolicy", "iam:PutRolePolicy", "iam:DeleteRolePolicy"]
          Effect   = "Allow"
          Resource = [for region in concat(["eu-north-1"], local.replication_regions) : "arn:aws:ecr:${region}:${local.account_id}:repository/${each.key}"]
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_iam_role_policy" "iam-policy" {
  name = "${local.atlantis_role_name}-iam-policy"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action = ["iam:GetRole", "iam:GetRolePolicy", "iam:ListRolePolicies", "iam:ListAttachedRolePolicies", "iam:PutRolePolicy", "iam:DeleteRolePolicy"]
          Effect = "Allow"
          Resource = [
            aws_iam_role.atlantis-role.arn,
          ]
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_ecr_repository" "team-repositories" {
  name     = each.key
  for_each = local.repositories
  depends_on      = [aws_iam_role_policy.ecr-policy]
}

resource "aws_ecr_repository_policy" "repo-settings" {
  for_each   = aws_ecr_repository.team-repositories
  repository = each.value.name
  policy = jsonencode({
    Version   = "2012-10-17",
    Statement = local.pull_image_permissions
  })
  depends_on      = [aws_iam_role_policy.ecr-policy]
}

provider "aws" {
  region = "ap-southeast-2"

  assume_role {
    role_arn = "arn:aws:iam::000000000000:role/custom-atlantis-ecr"
  }
}

Debug Output

https://gist.github.com/blakepettersson/58e77cf4e3018cb34e544d9f65f7e67a

Expected Behavior

This should work consistently on the first terraform apply.

Actual Behavior

This usually takes another terraform apply for this to work.

Steps to Reproduce

1. terraform apply

[chris-peterson]

To add some more context from my experiences with this issue --

The issue is specifically the sequence of ecr:CreateRepository followed by ecr:SetRepositoryPolicy (or ecr:PutLifecyclePolicy) within the same apply run. Subsequent runs of apply work; they pick up with a repository that was created in the first run, and the policy(s) are applied without issue.

This does not appear to be a timing issue as I cannot repro the issue using the AWS CLI, but something about the difference between a repository create and a read repository from state.

Capturing debug traces, the outputs between a failure and a success are nearly identical; i.e. the request payloads look the same, just one 403s while one 200s.

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!