Idempotency error: json objects read as strings cause resource Update calls when whitespace is detected

drewmullen commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Affected Resource(s)

awscc_networkmanager_core_network

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "awscc_networkmanager_global_network" "main" {
  description = "My Global Network"
}

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = <<EOF
{
  "version": "2021.12",
  "core-network-configuration": {
    "vpn-ecmp-support": false,
    "asn-ranges": [
      "64512-64555"
    ],
    "edge-locations": [
      {
        "location": "us-east-1",
        "asn": 64512
      }
    ]
  },
  "segments": [
    {
      "name": "shared",
      "require-attachment-acceptance": true
    },
    {
      "name": "prod",
      "require-attachment-acceptance": true
    },
    {
      "name": "finance",
      "require-attachment-acceptance": true
    },
    {
      "name": "hr",
      "require-attachment-acceptance": true
    },
    {
      "name": "vpn",
      "require-attachment-acceptance": true
    }
  ],
  "segment-actions": [
    {
      "action": "share",
      "mode": "attachment-route",
      "segment": "shared",
      "share-with": "*"
    }
  ],
  "attachment-policies": [
    {
      "rule-number": 100,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "shared"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "shared"
      }
    },
    {
      "rule-number": 200,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "prod"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "prod"
      }
    },
    {
      "rule-number": 300,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "finance"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "finance"
      }
    },
    {
      "rule-number": 400,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "hr"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "hr"
      }
    },
    {
      "rule-number": 500,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "vpn"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "vpn"
      }
    }
  ]
}
EOF
}

Expected Behavior

terraform apply (create)
terraform apply (no change)

Actual Behavior

terraform apply (create)

terraform apply (white space change detected)

# awscc_networkmanager_core_network.main will be updated in-place
~ resource "awscc_networkmanager_core_network" "main" {
    id                = "core-network-0ecdf681d32cba7b3"
  + owner_account     = (known after apply)
  ~ policy_document   = jsonencode( # whitespace changes

error

╷
│ Error: AWS SDK Go Service Operation Incomplete
│ 
│   with awscc_networkmanager_core_network.main,
│   on main.tf line 25, in resource "awscc_networkmanager_core_network" "main":
│   25: resource "awscc_networkmanager_core_network" "main" {
│ 
│ Waiting for Cloud Control API service UpdateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: Idempotency error.
│ Identical request made with different client token. (Service: NetworkManager, Status Code: 409, Request ID: <>). ErrorCode:
│ ResourceConflict

Steps to Reproduce

terraform apply

Important Factoids

References

0000

ewbankkit commented 2 years ago

After discussion with the relevant AWS service team it seems that the error is caused by attempting an update operation when there are no material changes to the resource. We need a way of suppressing the difference for JSON documents as we do in terraform-provider-aws. The immediate challenge is how to infer that such functionality is required given that the PolicyDocument property of the AWS::NetworkManager::CoreNetwork resource is of plain string type: https://github.com/hashicorp/terraform-provider-awscc/blob/39c2da41134bf303b046042544bce1690cfa7cc1/internal/service/cloudformation/schemas/AWS_NetworkManager_CoreNetwork.json#L18-L21

ewbankkit commented 2 years ago

A solution may be to mark that property as being of type object and define no nested properties for it. We would then need to change schema code generation to:

Generate a Terraform string type for the attribute, NOT a map type
Add a PlanModifier to suppress differences for equivalent JSON documents

ewbankkit commented 2 years ago

There are a number of such schema-less object properties, all of which seem like they would benefit from handling as JSON strings. For example:

generating Terraform resource code for "awscc_codeartifact_domain" from "../service/cloudformation/schemas/AWS_CodeArtifact_Domain.json" into "../aws/codeartifact/domain_resource_gen.go" and "../aws/codeartifact/domain_resource_gen_test.go"
PermissionsPolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_codeartifact_repository" from "../service/cloudformation/schemas/AWS_CodeArtifact_Repository.json" into "../aws/codeartifact/repository_resource_gen.go" and "../aws/codeartifact/repository_resource_gen_test.go"
PermissionsPolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_ec2_vpc_endpoint" from "../service/cloudformation/schemas/AWS_EC2_VPCEndpoint.json" into "../aws/ec2/vpc_endpoint_resource_gen.go" and "../aws/ec2/vpc_endpoint_resource_gen_test.go"
PolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_efs_file_system" from "../service/cloudformation/schemas/AWS_EFS_FileSystem.json" into "../aws/efs/file_system_resource_gen.go" and "../aws/efs/file_system_resource_gen_test.go"
FileSystemPolicy is of type object but has no schema
generating Terraform resource code for "awscc_stepfunctions_state_machine" from "../service/cloudformation/schemas/AWS_StepFunctions_StateMachine.json" into "../aws/stepfunctions/state_machine_resource_gen.go" and "../aws/stepfunctions/state_machine_resource_gen_test.go"
Definition is of type object but has no schema

ewbankkit commented 2 years ago

Even a suitable AttributePlanModifier is only half the story until #70 is addressed. The ReadResource path isn't affected by the PlanModifier so Terraform will still show an Objects have changed outside of Terraform message:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # awscc_networkmanager_core_network.main has been changed
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0c63c87f4f8751278"
      ~ policy_document   = jsonencode( # whitespace changes
            {
                attachment-policies        = [
                    {
                        action          = {
                            association-method = "constant"
                            segment            = "shared"
                        }
                        condition-logic = "or"
                        conditions      = [
                            {
                                key      = "segment"
                                operator = "equals"
                                type     = "tag-value"
                                value    = "shared"
                            },
                        ]
                        rule-number     = 1
                    },
                ]
                core-network-configuration = {
                    asn-ranges       = [
                        "64512-64555",
                    ]
                    edge-locations   = [
                        {
                            asn      = 64512
                            location = "us-east-1"
                        },
                    ]
                    vpn-ecmp-support = false
                }
                segment-actions            = [
                    {
                        action     = "share"
                        mode       = "attachment-route"
                        segment    = "shared"
                        share-with = "*"
                    },
                ]
                segments                   = [
                    {
                        description                   = "segment-for-shared-services"
                        name                          = "shared"
                        require-attachment-acceptance = true
                    },
                ]
                version                    = "2021.12"
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

ewbankkit commented 2 years ago

A workaround for this is to compose the Terraform jsondecode and jsonencode functions to produce a normalized JSON string:

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.main.json))
  tags              = local.terraform_tag
}

Now no diffs are shown:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-07142e587830550d4]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-07036ef298285217f]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

ewbankkit commented 2 years ago

If one of the policy values contains spaces I have noticed that the Cloud Control handler for the CoreNetwork resource is removing them:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-0560af1a3976af6d0]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-0682841d123a739d2]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # awscc_networkmanager_core_network.main has been changed
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0682841d123a739d2"
      ~ policy_document   = jsonencode(
          ~ {
              ~ segments                   = [
                  ~ {
                      ~ description                   = "Segment for shared services" -> "Segmentforsharedservices"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (4 unchanged elements hidden)
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # awscc_networkmanager_core_network.main will be updated in-place
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0682841d123a739d2"
      + owner_account     = (known after apply)
      ~ policy_document   = jsonencode(
          ~ {
              ~ segments                   = [
                  ~ {
                      ~ description                   = "Segmentforsharedservices" -> "Segment for shared services"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (4 unchanged elements hidden)
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.

tfhartmann commented 2 years ago

A workaround for this is to compose the Terraform jsondecode and jsonencode functions to produce a normalized JSON string:

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.main.json))
  tags              = local.terraform_tag
}

Now no diffs are shown:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-07142e587830550d4]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-07036ef298285217f]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Hi! I think I'm seeing this problem, even with the encode(decode()) work around. When I try to follow the example in this doc https://registry.terraform.io/providers/hashicorp%20%20/aws/latest/docs/guides/using-aws-with-awscc-provider

I get the following error :

│ Calling Cloud Control API service CreateResource operation returned: operation error CloudControl: CreateResource, https response error StatusCode: 400, RequestID:
│ bebae537-d031-4316-9194-6a2c0b3cf1de, api error ValidationException: Model validation failed (#/PolicyDocument: expected type: JSONObject, found: String)

This is on initial attempt to create the resources.

ewbankkit commented 2 years ago

@tfhartmann A one-off fix for awscc_networkmanager_core_network.policy_document was done in #537 which is planned to released in v0.25.0 of this provider, likely tomorrow.

tfhartmann commented 2 years ago

@tfhartmann A one-off fix for awscc_networkmanager_core_network.policy_document was done in #537 which is planned to released in v0.25.0 of this provider, likely tomorrow.

Awesome! Thanks!

joshtrutwin commented 1 year ago

Having similar issues with jsonencode on policies for

https://registry.terraform.io/providers/hashicorp/awscc/latest/docs/resources/opensearchserverless_access_policy https://registry.terraform.io/providers/hashicorp/awscc/latest/docs/resources/opensearchserverless_security_policy

resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy" {
  name        = "ce-test-aoss-encryption-policy"
  description = "Encryption Policy for CE Test Collection"
  type        = "encryption"

  policy = jsonencode({
    "Rules" : [
      {
        "Resource" : [
          "collection/ce-test-collection"
        ],
        "ResourceType" : "collection"
      }
    ],
    "AWSOwnedKey" : true
  })
}

perpetual drift followed by apply error:

awscc_opensearchserverless_security_policy.test_aoss_encryption_policy: Modifying... [id=encryption|ce-test-aoss-encryption-policy]
╷
│ Error: AWS SDK Go Service Operation Incomplete
│ 
│   with awscc_opensearchserverless_security_policy.test_aoss_encryption_policy,
│   on opensearch.tf line 31, in resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy":
│   31: resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy" {
│ 
│ Waiting for Cloud Control API service UpdateResource operation completion
│ returned: waiter state transitioned to FAILED. StatusMessage: Invalid
│ request provided: UpdateSecurityPolicyRequest(Description=Encryption Policy
│ for CE Test Collection, Name=ce-test-aoss-encryption-policy,
│ Policy={"AWSOwnedKey":true,"Rules":[{"Resource":["collection/ce-test-collection"],"ResourceType":"collection"}]},
│ PolicyVersion=MTY3NzYyMjUyMTgyNl8x, Type=encryption). ErrorCode:
│ InvalidRequest
╵
Operation failed: failed running terraform apply (exit 1)

Thanks

joshtrutwin commented 1 year ago

Cloudtrail shows:

    "eventSource": "aoss.amazonaws.com",
    "eventName": "UpdateSecurityPolicy",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "No changes detected in policy or policy description",
    "requestParameters": {
        "type": "encryption",
        "name": "ce-test-aoss-encryption-policy",
        "policyVersion": "MTY3NzYyMjUyMTgyNl8x",
        "description": "Encryption Policy for CE Test Collection",
        "policy": "{\"AWSOwnedKey\":true,\"Rules\":[{\"Resource\":[\"collection/ce-test-collection\"],\"ResourceType\":\"collection\"}]}",
        "clientToken": "<snip>"
    },

meraj-kashi commented 2 months ago

Same issue in awscc_logs_account_policy:

resource "awscc_logs_account_policy" "cloudwatch_account_level_subscription" {
  policy_name = var.log_policy_name
  policy_type = var.log_policy_type
  scope       = var.log_policy_scope
  #selection_criteria = var.log_policy_type == "SUBSCRIPTION_FILTER_POLICY" ? try(var.log_selection_criteria, null) : null
  policy_document = jsonencode(jsondecode(<<EOF
{
  "DestinationArn": "${var.aws_cloudwatch_logs_destination_arn}",
  "RoleArn": "${aws_iam_role.cloudwatch_account_level_subscription_role[0].arn}",
  "FilterPattern": "${var.log_filter_pattern}",
  "Distribution": "${var.log_distribution}"
}
EOF
  ))
}

Terraform plan shows diff everytime due to whitespace changes:

 # awscc_logs_account_policy.cloudwatch_account_level_subscription[0] will be updated in-place
resource "awscc_logs_account_policy" "cloudwatch_account_level_subscription" {
        id                 = "634103029566|SUBSCRIPTION_FILTER_POLICY|NTAccountLevelSubscriptionPolicy"
      ~ policy_document    = jsonencode( # whitespace changes
            {

quixoticmonk commented 2 months ago

@meraj-kashi in your specific case, have you tried with the jsonencode() alone ?

meraj-kashi commented 2 months ago

Hi! Yes, first I tried using jsonencode() alone. After encountering the issue, I searched for any bugs or reported issues that might have caused it. Then, I tried the suggested workaround (jsonencode(jsondecode)). In my case, I found out that the AWS provider recently added this resource, so I switched to the AWS provider resource: aws_cloudwatch_log_account_policy.

Sammmmm commented 1 month ago

I'm having what appears to be the same issue -- inline policy, jsonencode


resource "awscc_iam_role" "iot_silo_kinesis_s3_role" {
  role_name = "iot-silo-kinesis-s3-role"
  assume_role_policy_document = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "sts:AssumeRole"
        ],
        "Principal" : {
          "Service" : [
            "firehose.amazonaws.com"
          ]
        }
      }
    ]
  })
}

jlamande commented 1 month ago

Same issue (perpetual " # whitespace changes" ) in the policy field for ECR Creation templates :

awscc_ecr_repository_creation_template { 
  ...
  repository_policy =
  ...
}

tried jsonencode(jsondecode(data.aws_iam_policy_document.template_repo_policy.json)), heredoc string, ... but still the same.

hashicorp / terraform-provider-awscc