Unable to change MSK broker storage size with Terraform

[visah]

Description

After provisioning a new MSK cluster with fixed storage size it's not possible to change the storage size with Terraform. It seems like Terraform is unable to detect the change in the broker storage size.

• [x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version : 2.5 (Latest in Terraform registry when creating the issue.)

• Terraform version: v1.5.7 and also latest v1.8.5 on linux_amd64

• Provider version(s): provider registry.terraform.io/hashicorp/aws v5.54.1 provider registry.terraform.io/hashicorp/random v3.6.2

Reproduction Code [Required]

module "msk" {
  source  = "terraform-aws-modules/msk-kafka-cluster/aws"
  version = "2.5.0"

  name                      = "dev"
  broker_node_instance_type = "kafka.t3.small"
  kafka_version             = "3.5.1"
  number_of_broker_nodes    = 2

  client_authentication = {
    sasl = { iam = true }
  }

  broker_node_client_subnets = [
...
  ]

  enable_storage_autoscaling = false
  broker_node_storage_info = {
    ebs_storage_info = {
      volume_size = 5
      }
    }
}

Steps to reproduce the behavior:

1. Create the MSK cluster with fixed storage space
2. Try to increase the storage space

Expected behavior

Terraform plan should show that it would change the EBS volume size from 5 GB to 20 GB.

Actual behavior

Terraform plan shows "No changes. Your infrastructure matches the configuration.".

[bryantbiggs]

this module is set up to support autoscaling of the storage, which means that direct manipulating of the storage is ignored by Terraform to avoid conflicts https://github.com/terraform-aws-modules/terraform-aws-msk-kafka-cluster/blob/0496baa7c1e193ef4e22065d7f9cdc438744f797/main.tf#L164

If you need to do it through Terraform, you would need to do something like this https://github.com/bryantbiggs/eks-desired-size-hack

[visah]

Thanks for the quick reply to this. I didn't realize that the module works that way. So it's set to ignore manual changes to the EBS volume size on purpose. Maybe it's then best to just use the auto-scaling.

[bryantbiggs]

So it's set to ignore manual changes to the EBS volume size on purpose

I wouldn't call them manual changes - its designed to allow users to specify an initial storage volume size, but after the initial creation with Terraform and that initial value, Terraform will ignore any further changes to the storage volume size. This allows other external parties (i.e. - autoscaling) to adjust the storage parameter without causing conflicts with Terraform

So in theory, you could perform a manual change if desired by changing the storage volume size through the AWS console, or via an awscli API call and it won't affect Terraform

[visah]

So in theory, you could perform a manual change if desired by changing the storage volume size through the AWS console, or via an awscli API call and it won't affect Terraform

This is not actually how it works or at least I faced an issue after growing the broker storage via AWS console. Terraform was then giving me this error.

# module.msk.module.msk.aws_msk_cluster.this[0] will be updated in-place
  ~ resource "aws_msk_cluster" "this" {
        id                           = "arn:aws:kafka:us-east-1:xxxxxxxxx:cluster/shared/4ee8eeba-7242-44be-b9a3-0f456b7405b0-7"
        tags                         = {}
        # (12 unchanged attributes hidden)

      ~ broker_node_group_info {
            # (4 unchanged attributes hidden)

          ~ storage_info {
              ~ ebs_storage_info {
                    # (1 unchanged attribute hidden)

                  - provisioned_throughput {
                      - enabled           = false -> null
                      - volume_throughput = 0 -> null
                    }
                }
            }

            # (1 unchanged block hidden)
        }

        # (6 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

  module.msk.module.msk.aws_msk_cluster.this[0]: Modifying... [id=arn:aws:kafka:us-east-1:xxxxxxxxxxx:cluster/shared/4ee8eeba-7242-44be-b9a3-0f456b7405b0-7]
╷
│ Error: updating MSK Cluster (arn:aws:kafka:us-east-1:xxxxxxxxxxx:cluster/shared/4ee8eeba-7242-44be-b9a3-0f456b7405b0-7) broker storage: operation error Kafka: UpdateBrokerStorage, https response error StatusCode: 400, RequestID: 9fd0e8fe-c152-4138-8f07-4551e28e2888, BadRequestException: The request does not include any updates to the EBS volumes of the cluster. Verify the request, then try again.
│ 
│   with module.msk.module.msk.aws_msk_cluster.this[0],
│   on .terraform/modules/msk.msk/main.tf line 5, in resource "aws_msk_cluster" "this":
│    5: resource "aws_msk_cluster" "this" {
│ 
╵

[bryantbiggs]

hmm, looks like a bug in the provider https://github.com/hashicorp/terraform-provider-aws/issues/20327#issuecomment-1978370103

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days