[Bug]: Removing tls/sasl block will not turn off corresponding authentication method on AWS MSK cluster

[quercusilvam]

Terraform Core Version

1.3.1

AWS Provider Version

4.63.0

Affected Resource(s)

aws_msk_cluster

Expected Behavior

After adding sasl or tls authentication for MSK cluster, removing them should also turn off those settings on AWS MSK. As described in documentation, defaults should be false which should turn off removed authentication method.

Actual Behavior

Removing sasl or tls config in client_authentication block tries to set corresponding options from true to null but this is not working. As a result AWS MSK authentication options still includes removed method(s).

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_msk_cluster" "test" {

  cluster_name           = "test"
  kafka_version          = "3.2.0"
  number_of_broker_nodes = 2

  broker_node_group_info {
    instance_type   = kafka.t3.small
    client_subnets  = ["subnet-1", "subnet-2"]
    security_groups = ["sg-123456789"]
  }

  client_authentication {
    tls {
      certificate_authority_arns = "arn:aws:acm-pca:eu-west-1:XXXXXX:certificate-authority/123456789"
    }
    sasl {
      iam = true
    }
    unauthenticated = false
  }
}

Steps to Reproduce

Just remove sasl block

resource "aws_msk_cluster" "test" {

  cluster_name           = "test"
  kafka_version          = "3.2.0"
  number_of_broker_nodes = 2

  broker_node_group_info {
    instance_type   = kafka.t3.small
    client_subnets  = ["subnet-1", "subnet-2"]
    security_groups = ["sg-123456789"]
  }

  client_authentication {
    tls {
      certificate_authority_arns = "arn:aws:acm-pca:eu-west-1:XXXXXX:certificate-authority/123456789"
    }
    unauthenticated = false
  }
}

Debug Output

Terraform plan shows those changes (set to nulls instead of false). But this will not work

~ client_authentication {
    # (1 unchanged attribute hidden)

  - sasl {
      - iam   = true -> null
      - scram = false -> null
    }

    # (1 unchanged block hidden)
}

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[quercusilvam]

Workaround: You can change value to explicit false. This way it works, cluster will be updated.

client_authentication {
    tls {
      certificate_authority_arns = "arn:aws:acm-pca:eu-west-1:XXXXXX:certificate-authority/123456789"
    }
    sasl {
      iam = false
    }
    unauthenticated = false
  }

Unfortunately if you remove afterwards the sasl block you will get never-ending updates of a cluster at next terraform apply as described here: https://github.com/hashicorp/terraform-provider-aws/issues/24914#issuecomment-1509301036

[quercusilvam]

I've also tested what will happen if I set false for not used authentication methods (I've simulated default value described in provider documentation).

  client_authentication {
    tls {
      certificate_authority_arns = local.pca_arn
    }
    sasl {
      iam   = false
      scram = false
    }
    unauthenticated = false
  }

And it is working correctly - I can create MSK cluster, in state file iam/scram methods are set to false. Terraform plan after creation report no changes.

In my opinion this could be valid fix - set false values if blocks are missing (not null as is this now).