Constant json changes with aws_redshift_parameter_group, whitespace issue?

mtekel commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.19

provider.aws v2.47.0

Affected Resource(s)

aws_redshift_parameter_group

and potentially many more

Terraform Configuration Files

See terraform plan output below. I have copy-pasted that exact config.

Debug Output

Bits and pieces with trace enabled. It figures out there is a diff, but doesn't show the actual diff:

2020/02/14 11:51:14 [TRACE] EvalReadState: reading state for aws_redshift_parameter_group.main
2020/02/14 11:51:14 [TRACE] UpgradeResourceState: schema version of aws_redshift_parameter_group.main is still 0; calling provider "registry.terraform.io/-/aws" for any other minor fixups
...
2020/02/14 11:51:14 [TRACE] Re-validating config for "aws_redshift_parameter_group.main"
...
2020/02/14 11:51:14 [TRACE] EvalWriteState: writing current state object for aws_redshift_parameter_group.main
2020/02/14 11:51:14 [TRACE] EvalWriteDiff: recorded Update change for aws_redshift_parameter_group.main
2020/02/14 11:51:14 [TRACE] [walkPlan] Exiting eval tree: aws_redshift_parameter_group.main
2020/02/14 11:51:14 [TRACE] vertex "aws_redshift_parameter_group.main": visit complete
2020/02/14 11:51:14 [TRACE] vertex "aws_redshift_parameter_group.main": dynamic subgraph completed successfully
2020/02/14 11:51:14 [TRACE] vertex "aws_redshift_parameter_group.main": visit complete

Expected Behavior

No diff.

Actual Behavior

TF wants to diff:

# aws_redshift_parameter_group.main will be updated in-place
~ resource "aws_redshift_parameter_group" "main" {
      arn         = "arn:aws:redshift:us-east-1:1234556789:parametergroup:main"
      description = "main"
      family      = "redshift-1.0"
      id          = "main"
      name        = "main"
      tags        = {}

    + parameter {
        + name  = "wlm_json_configuration"
        + value = jsonencode(
              [
                + {
                    + max_execution_time    = 0
                    + memory_percent_to_use = 30
                    + query_concurrency     = 4
                    + query_group           = [
                        + "admin_group_1",
                        + "admin_group_2",
                      ]
                    + query_group_wild_card = 0
                    + user_group            = [
                        + "admin_group",
                        + "test_group",
                      ]
                    + user_group_wild_card  = 0
                  },
                + {
                    + max_execution_time    = 0
                    + memory_percent_to_use = 69
                    + query_concurrency     = 8
                    + query_group           = [
                        + "analytics_group_1",
                        + "analytics_group_2",
                      ]
                    + query_group_wild_card = 0
                    + user_group            = [
                        + "analytics_group",
                      ]
                    + user_group_wild_card  = 0
                  },
                + {
                    + max_execution_time    = 0
                    + memory_percent_to_use = 1
                    + query_concurrency     = 1
                    + query_group           = []
                    + query_group_wild_card = 0
                    + user_group            = []
                    + user_group_wild_card  = 0
                  },
                + {
                    + short_query_queue = true
                  },
              ]
          )
      }
    - parameter {
        - name  = "wlm_json_configuration" -> null
        - value = jsonencode(
              [
                - {
                    - max_execution_time    = 0
                    - memory_percent_to_use = 30
                    - query_concurrency     = 4
                    - query_group           = [
                        - "admin_group_1",
                        - "admin_group_2",
                      ]
                    - query_group_wild_card = 0
                    - user_group            = [
                        - "admin_group",
                        - "test_group",
                      ]
                    - user_group_wild_card  = 0
                  },
                - {
                    - max_execution_time    = 0
                    - memory_percent_to_use = 69
                    - query_concurrency     = 8
                    - query_group           = [
                        - "analytics_group_1",
                        - "analytics_group_2",
                      ]
                    - query_group_wild_card = 0
                    - user_group            = [
                        - "analytics_group",
                      ]
                    - user_group_wild_card  = 0
                  },
                - {
                    - max_execution_time    = 0
                    - memory_percent_to_use = 1
                    - query_concurrency     = 1
                    - query_group           = []
                    - query_group_wild_card = 0
                    - user_group            = []
                    - user_group_wild_card  = 0
                  },
                - {
                    - short_query_queue = true
                  },
              ]
          ) -> null
      }
  }

Steps to Reproduce

Import resource from AWS: terraform import aws_redshift_parameter_group.main main
Copy-paste what TF wants to remove (the parameter group), remove -> null and minuses and you get hcl
terraform plan

...there is no actual visible diff between the + and the - resource in the plan, they are exactly same

I have also tried to get the json directly from AWS and put it as 1. Herefile value, 2. Into file and read it (value = file("redshift-params.json"). Still same issue.

Important Factoids

The fix was to get actual value from statefile after import and copy-paste it from there:

        value = "[{\"user_group_wild_card\": 0, \"query_group_wild_card\": 0, \"max_execution_time\": 0, \"query_group\": [\"admin_group_1\", \"admin_group_2\"], \"user_group\": [\"admin_group\", \"test_group\"], \"query_concurrency\": 4, \"memory_percent_to_use\": 30}, {\"user_group_wild_card\": 0, \"query_group_wild_card\": 0, \"max_execution_time\": 0, \"query_group\": [\"analytics_group_1\", \"analytics_group_2\"], \"user_group\": [\"analytics_group\"], \"query_concurrency\": 8, \"memory_percent_to_use\": 69}, {\"user_group_wild_card\": 0, \"query_group_wild_card\": 0, \"max_execution_time\": 0, \"query_group\": [], \"user_group\": [], \"query_concurrency\": 1, \"memory_percent_to_use\": 1}, {\"short_query_queue\": true}]"

Note that json is now squashed to a single line and quotes are escaped.

References

https://github.com/hashicorp/terraform/issues/21473 Perhaps this as well: https://github.com/hashicorp/terraform/issues/23928

Based on comments, it seems that json is somehow not normalised for aws_redshift_parameter_group and then you get never ending whitespace diffs. Even worse - if you apply and then refresh, AWS might return a bit different (whitespace wise) json and TF might want to apply again.

ewbankkit commented 4 years ago

Similar:

https://github.com/terraform-providers/terraform-provider-aws/issues/7865

mtekel commented 4 years ago

Same issues with aws_s3_bucket_policy. Another level of complication is when you want to do some interpolation in the policy itself. In that case, workarounds might not work, as interpolation could be changing json processing/result in a way that it would never be the same as one that was imported.

Ideally, because this goes to tfstate, terraform should have some function in core that it pushes all data through. This way there would be no changes on import vs plan.

justinretzolk commented 2 years ago

Hey @mtekel 👋 Thank you for taking the time to file this issue. Given that there's been a number of AWS provider releases since you initially filed it, can you confirm whether you're still experiencing this behavior?

dtsveitel commented 2 years ago

I can confirm still seeing the issue with 1.0.9 on IAM policy content:

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_iam_role.jenkins_org_task_role has been changed
  ~ resource "aws_iam_role" "jenkins_org_task_role" {
      ~ assume_role_policy    = jsonencode( # whitespace changes
            {
                Statement = [
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::xxxxxxxxxxxx:role:role/ADFS-OpsAdmin"
                            Service = "ecs-tasks.amazonaws.com"
                        }
                        Sid       = ""
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        id                    = "ecp-cicd-task-org"
        name                  = "ecp-cicd-task-org"
        tags                  = {}
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_iam_role.ecs_execution_role has been changed
  ~ resource "aws_iam_role" "ecs_execution_role" {
      ~ assume_role_policy    = jsonencode( # whitespace changes
            {
                Statement = [
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::xxxxxxxxxxxx:role:role/ADFS-OpsAdmin"
                            Service = "ecs-tasks.amazonaws.com"
                        }
                        Sid       = ""
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        id                    = "ecp-cicd-ecs-execution"
        name                  = "ecp-cicd-ecs-execution"
        tags                  = {}
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # aws_iam_role.jenkins_selfupdate_task_role has been changed
  ~ resource "aws_iam_role" "jenkins_selfupdate_task_role" {
      ~ assume_role_policy    = jsonencode( # whitespace changes
            {
                Statement = [
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::xxxxxxxxxxxx:role/ADFS-OpsAdmin"
                            Service = "ecs-tasks.amazonaws.com"
                        }
                        Sid       = ""
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        id                    = "ecp-cicd-task-selfupdate"
        name                  = "ecp-cicd-task-selfupdate"
        tags                  = {}
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

luisamador commented 2 years ago

We hit this same issue too. I'm surprised it has not been fixed yet since then.

gmocquet commented 2 years ago

It's the same from my side, with latest Terraform version (1.1.9) and latest terraform-provider-aws/hashicorp/aws (4.13.0).

Below, the Terraform source code:

data "aws_iam_policy_document" "sagemaker_execution_role" {
  version = "2012-10-17"

  statement {
    sid     = "apigateway4astronomer"
    actions = ["sts:AssumeRole"]

    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${var.account_id}:user/astronomer-secret"]
    }

    principals {
      type        = "Service"
      identifiers = ["apigateway.amazonaws.com"]
    }
  }

  statement {
    sid     = "apigateway4github"
    actions = ["sts:AssumeRole"]

    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::yyyyy:user/github-datascience"]
    }

    principals {
      type        = "Service"
      identifiers = ["apigateway.amazonaws.com"]
    }
  }

  statement {
    sid     = "sagemaker4astronomer"
    actions = ["sts:AssumeRole"]

    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${var.account_id}:user/astronomer-secret"]
    }

    principals {
      type        = "Service"
      identifiers = ["sagemaker.amazonaws.com"]
    }
  }

  statement {
    sid     = "sagemaker4github"
    actions = ["sts:AssumeRole"]

    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::yyyy:user/github-datascience"]
    }

    principals {
      type        = "Service"
      identifiers = ["sagemaker.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "sagemaker_execution_role" {
  name               = local.common_name
  description        = "Sagemaker IAM execution role - ${var.common_description}"
  assume_role_policy = data.aws_iam_policy_document.sagemaker_execution_role.json
  managed_policy_arns = [
    aws_iam_policy.sagemaker_execution_role.arn,
    "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs",
    "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess",
  ]
  max_session_duration = 43200
}

Note: I don't use array / list of identifiers due to Terraform bug: the order is not the same, so, Terraform see changes during the apply.

Below, the Terraform log from the apply command:

module.rds.module.kpis.postgresql_grant.airflow_database: Refreshing state... [id=airflow_kpis_database]
module.rds.module.kpis.postgresql_grant.airflow_schema: Refreshing state... [id=airflow_kpis_public_schema]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.sagemaker.aws_iam_role.sagemaker_execution_role has changed
  ~ resource "aws_iam_role" "sagemaker_execution_role" {
      ~ assume_role_policy    = jsonencode( # whitespace changes
            {
                Statement = [
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::xxxxxx:user/astronomer-secret"
                            Service = "apigateway.amazonaws.com"
                        }
                        Sid       = "apigateway4astronomer"
                    },
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::yyyyy:user/github-datascience"
                            Service = "apigateway.amazonaws.com"
                        }
                        Sid       = "apigateway4github"
                    },
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::xxxxxx:user/astronomer-secret"
                            Service = "sagemaker.amazonaws.com"
                        }
                        Sid       = "sagemaker4astronomer"
                    },
                    {
                        Action    = "sts:AssumeRole"
                        Effect    = "Allow"
                        Principal = {
                            AWS     = "arn:aws:iam::yyyyyy:user/github-datascience"
                            Service = "sagemaker.amazonaws.com"
                        }
                        Sid       = "sagemaker4github"
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        id                    = "kp-sagemaker-datascience-dev-common"
        name                  = "kp-sagemaker-datascience-dev-common"
        tags                  = {}
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│ 
│   on modules/secrets_manager/variables.tf line 2, in terraform:
│    2:   experiments = [module_variable_optional_attrs]
│ 
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│ 
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│ 
│ (and 3 more similar warnings elsewhere)
╵

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:
[...]

hashicorp / terraform-provider-aws