aws_iam_role_policy Error: Provider produced inconsistent final plan

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

aws_iam_role_policy

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  region = "eu-central-1"
}

data "aws_iam_policy_document" "assume_role_policy" {
  version = "2012-10-17"
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "lambda_role" {
  name                = "Lambda-Role"
  description         = "Lambda Role"
  path                = "/"
  assume_role_policy  = data.aws_iam_policy_document.assume_role_policy.json
  managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"]
}

data "aws_iam_policy_document" "lambda_policy_document" {
  version = "2012-10-17"
  statement {
    actions = [
      "logs:PutRetentionPolicy"
    ]
    effect = "Allow"
    resources = [
      "arn:aws:logs:*:*:log-group:*"
    ]
  }
}

resource "aws_iam_policy" "lambda_policy" {
  name        = "Lambda-Policy"
  description = "Lambda Policy"
  policy      = data.aws_iam_policy_document.lambda_policy_document.json
}

resource "aws_iam_role_policy" "lambda_role_policy" {
  name   = "Lambda-Role-Policy"
  role   = aws_iam_role.lambda_role.id
  policy = aws_iam_policy.lambda_policy.policy
}

Debug Output

Expected Behavior

The tf should apply without errors

Actual Behavior

It does not deploy, it throws the error;

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_iam_role_policy.lambda_role_policy to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .policy: was cty.StringVal("{\n  \"Version\":
│ \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"logs:PutRetentionPolicy\",\n      \"Resource\": \"arn:aws:logs:*:*:log-group:*\"\n    }\n  ]\n}"), but now
│ cty.StringVal("{\"Statement\":[{\"Action\":\"logs:PutRetentionPolicy\",\"Effect\":\"Allow\",\"Resource\":\"arn:aws:logs:*:*:log-group:*\",\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Steps to Reproduce

terraform apply

Able to reproduce this issue with code like

resource "aws_iam_policy" "blah" {
  name   = "${terraform.workspace}_blah"
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParametersByPath",
                "ssm:GetParameter"
            ],
            "Resource": [
                ...
            ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:BatchGetImage"
          ],
          "Resource": "*"
        }
    ]
}

EOF

}

resource "aws_iam_role_policy" "blah" {
  name   = "${terraform.workspace}_blah"
  role   = aws_iam_role.blah.id
  policy = aws_iam_policy.blah.policy
}

Running plan and apply separately, I think it saves the plan file and then passes it to the apply.

Getting:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_iam_role_policy.blah to include new
│ values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .policy: was cty.StringVal("{\n    \"Version\": \"2012-10-17\",\n
│ \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n
│ \"Action\": [\n                \"ssm:GetParametersByPath\",\n
│ \"ssm:GetParameter\"\n            ],\n            \"Resource\": [\n...]\n
│ },\n        {\n          \"Effect\": \"Allow\",\n          \"Action\": [\n
│ \"ecr:GetAuthorizationToken\",\n
│ \"ecr:BatchCheckLayerAvailability\",\n
│ \"ecr:GetDownloadUrlForLayer\",\n            \"ecr:BatchGetImage\"\n
│ ],\n          \"Resource\": \"*\"\n        }\n    ]\n}\n\n"), but now
│ cty.StringVal("{\"Statement\":[{\"Action\":[\"ssm:GetParametersByPath\",\"ssm:GetParameter\"],\"Effect\":\"Allow\",\"Resource\":[...]},{\"Action\":[\"ecr:GetAuthorizationToken\",\"ecr:BatchCheckLayerAvailability\",\"ecr:GetDownloadUrlForLayer\",\"ecr:BatchGetImage\"],\"Effect\":\"Allow\",\"Resource\":\"*\"}],\"Version\":\"2012-10-17\"}").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

I cleaned up these two strings into properly formatted json and they are identical OBJECTS, but the strings are different because of ordering of keys among other things.

According to

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy.html#policy

policy - (Required) The inline policy document. This is a JSON formatted string.

When we do

resource "aws_iam_role_policy" "blah" {
  name   = "${terraform.workspace}_blah"
  role   = aws_iam_role.blah_ec2.id
  policy = aws_iam_policy.blah.policy
}

the policy = aws_iam_policy.blah.policy part is passing a JSON string..

I had the idea of trying to canonicalize the JSON (sort of object keys and sort the values within any arrays that are technically sets, not lists)

I thought of trying to run something like the following in a shell script null_resource:

jq --sort-keys '{
  Version: .Version,
  Statement: [.Statement[] | {
    Effect: .Effect,
    Action: (if (.Action | type) == "array" then (.Action | sort) else .Action end),
    Resource: (if (.Resource | type) == "array" then (.Resource | sort) else .Resource end)
  }] | sort_by("\(.Effect).\((if (.Action | type) == "array" then (.Action | sort) else .Action end))\((if (.Resource | type) == "array" then (.Resource | sort) else .Resource end))")
}'

Of course that only covers a subset of the policy JSON schema, and IDK if it will even work, but thought I'd share the idea. if we give it a try i'll post an update.

hashicorp / terraform-provider-aws