Eventual consistency error with ECR `GetRepositoryPolicy` / `SetRepositoryPolicy` (?)

[blakepettersson]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.8 on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.29.0

Affected Resource(s)

• aws_ecr_repository_policy
• aws_iam_role_policy

Terraform Configuration Files

data "aws_caller_identity" "current" {}

locals {
  atlantis_role_name = "custom-atlantis-ecr"
  account_id           = data.aws_caller_identity.current.account_id
  replication_regions  = ["eu-central-1", "us-east-1", "us-east-2", "ap-southeast-2", "eu-west-1"]
  repositories = {
    "11" = ["arn:aws:iam::000000000000:root"]
    "22" = ["arn:aws:iam::000000000000:root"]
  }
  pull_image_permissions = {
    Sid    = "PullImages",
    Effect = "Allow",
    Action = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:GetRepositoryPolicy",
      "ecr:DescribeRepositories",
      "ecr:ListImages",
      "ecr:DescribeImages",
      "ecr:BatchGetImage",
      "ecr:DescribeImageScanFindings"
    ],
    Principal = { AWS = "*" }
  }
}

resource "aws_iam_role" "atlantis-role" {
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : ["*"]
        },
        "Action" : "sts:AssumeRole"
      }
    ]
  })

  name                = local.atlantis_role_name
  managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonVPCReadOnlyAccess"]
}

resource "aws_iam_role_policy" "create-repository-policy" {
  name = "${local.atlantis_role_name}-policy"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action   = ["ecr:CreateRepository", "ecr:DescribeRegistry", "ecr:GetRegistryScanningConfiguration", "ecr:DescribeRepositories", "ecr:ListTagsForResource"]
          Effect   = "Allow"
          Resource = "*"
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_iam_role_policy" "ecr-policy" {
  name     = "${local.atlantis_role_name}-${each.key}-policy"
  for_each = local.repositories
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action   = ["ecr:DeleteRepository", "ecr:DeleteRepositoryPolicy", "ecr:SetRepositoryPolicy", "iam:PutRolePolicy", "iam:DeleteRolePolicy"]
          Effect   = "Allow"
          Resource = [for region in concat(["eu-north-1"], local.replication_regions) : "arn:aws:ecr:${region}:${local.account_id}:repository/${each.key}"]
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_iam_role_policy" "iam-policy" {
  name = "${local.atlantis_role_name}-iam-policy"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          Action = ["iam:GetRole", "iam:GetRolePolicy", "iam:ListRolePolicies", "iam:ListAttachedRolePolicies", "iam:PutRolePolicy", "iam:DeleteRolePolicy"]
          Effect = "Allow"
          Resource = [
            aws_iam_role.atlantis-role.arn,
          ]
        }
      ]
    }
  )
  role = aws_iam_role.atlantis-role.name
}

resource "aws_ecr_repository" "team-repositories" {
  name     = each.key
  for_each = local.repositories
  depends_on      = [aws_iam_role_policy.ecr-policy]
}

resource "aws_ecr_repository_policy" "repo-settings" {
  for_each   = aws_ecr_repository.team-repositories
  repository = each.value.name
  policy = jsonencode({
    Version   = "2012-10-17",
    Statement = local.pull_image_permissions
  })
  depends_on      = [aws_iam_role_policy.ecr-policy]
}

provider "aws" {
  region = "ap-southeast-2"

  assume_role {
    role_arn = "arn:aws:iam::000000000000:role/custom-atlantis-ecr"
  }
}

Debug Output

https://gist.github.com/blakepettersson/58e77cf4e3018cb34e544d9f65f7e67a

Expected Behavior

This should work consistently on the first terraform apply.

Actual Behavior

This usually takes another terraform apply for this to work.

Steps to Reproduce

1. terraform apply

[chris-peterson]

To add some more context from my experiences with this issue --

The issue is specifically the sequence of ecr:CreateRepository followed by ecr:SetRepositoryPolicy (or ecr:PutLifecyclePolicy) within the same apply run. Subsequent runs of apply work; they pick up with a repository that was created in the first run, and the policy(s) are applied without issue.

This does not appear to be a timing issue as I cannot repro the issue using the AWS CLI, but something about the difference between a repository create and a read repository from state.

Capturing debug traces, the outputs between a failure and a success are nearly identical; i.e. the request payloads look the same, just one 403s while one 200s.