[Bug]: aws_msk_cluster bootstrap_brokers_sasl_iam permadiff

[GergelyKalmar]

Terraform Core Version

1.2.6

AWS Provider Version

5.35.0

Affected Resource(s)

• aws_msk_cluster

Expected Behavior

The bootstrap_brokers_sasl_iam attribute should remain consistent between plans.

Actual Behavior

The bootstrap_brokers_sasl_iam is changing on every plan when having more than 3 brokers:

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # aws_msk_cluster.this has changed
  ~ resource "aws_msk_cluster" "this" {
      ~ bootstrap_brokers_sasl_iam   = "b-1.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-4.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-6.<redacted>.kafka.us-east-1.amazonaws.com:9098" -> "b-4.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-5.<redacted>.c8.kafka.us-east-1.amazonaws.com:9098,b-6.<redacted>.kafka.us-east-1.amazonaws.com:9098"
    }

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

N/A

Steps to Reproduce

1. Create an aws_msk_cluster resource with 6 brokers.
2. Run a terraform apply.
3. Run the terraform apply again.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jenna-foghorn]

Also experiencing this...

...in only one of our three accounts where the same terraform code is deployed. I would only expect brokers to be updated if something of substance changes, like authentication or encryption? This trickles down to each MSK Connect connector that has kafka.bootstrap.servers as part of the connector config.

I'm guessing that means that the MSK Cluster updates it's brokers... and my MSK Connect connector is busted until someone runs an apply again?

Since this occurs every time I'm running apply, these brokers are flapping in the wind!!

Each and EVERY apply shows:

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.msk.aws_msk_cluster.this has changed
  ~ resource "aws_msk_cluster" "this" {
      ~ bootstrap_brokers_sasl_iam   = "b-2.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-3.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-4.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098" -> "b-2.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-4.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098,b-5.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9098"
      ~ bootstrap_brokers_sasl_scram = "b-2.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096,b-3.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096,b-4.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096" -> "b-2.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096,b-4.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096,b-5.us1standalone.<redacted>.kafka.us-east-1.amazonaws.com:9096"
        id                           = "arn:aws:kafka:us-east-1:***:cluster/us1-standalone/<redacted>"
        tags                         = {
            "CostCenter"   = "Us1"
            "Environment"  = "us1"
            "Workload"     = "msk"
            "map-migrated" = "mig24333"
        }
        # (11 unchanged attributes hidden)

        # (6 unchanged blocks hidden)
    }

[GergelyKalmar]

It generally did not cause problems for our connectors (like re-creations or anything) or other resources. Nonetheless, nobody likes an unnecessary permadiff. I think this might be actually a problem with the AWS API itself being non-deterministic.

[GergelyKalmar]

Well, now I see that it did not cause issues because we're explicitly ignoring changes to the bootstrap workers in our connectors :upside_down_face::

  lifecycle {
    ignore_changes = [
      # See https://github.com/hashicorp/terraform-provider-aws/issues/35696
      kafka_cluster[0].apache_kafka_cluster[0].bootstrap_servers,
    ]