vrchen commented 9 months ago

Description

Please add Graviton instance types to the MSK resource (aws_msk_cluster). I'm on version 5.30.0 and am getting this error when attempting to use a graviton instance for my MSK cluster.

{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "8ac31d68-17ac-42f0-b698-e8bbbb1793b3"
  },
  InvalidParameter: "instanceType",
  Message_: "Unsupported InstanceType specified. Valid values: [kafka.m5.12xlarge, kafka.m5.16xlarge, kafka.m5.24xlarge, kafka.m5.2xlarge, kafka.m5.4xlarge, kafka.m5.8xlarge, kafka.m5.large, kafka.m5.xlarge, kafka.t3.small]"
}

my plan:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  # module.tiered_spike_kafka[0].aws_msk_cluster.msk will be updated in-place
  ~ resource "aws_msk_cluster" "msk" {
        id                           = "REDACTED"
        tags                         = {
            "Name"             = "REDACTED"
            "business_unit"    = "REDACTED"
            "cell_id"          = "REDACTED"
            "environment"      = "dev"
            "environment_type" = "dev"
            "resource_owner"   = "REDACTED"
            "service_name"     = "REDACTED"
        }
        # (12 unchanged attributes hidden)
      ~ broker_node_group_info {
          ~ instance_type   = "kafka.m5.large" -> "kafka.m7g.large"
            # (3 unchanged attributes hidden)
            # (2 unchanged blocks hidden)
        }
        # (5 unchanged blocks hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Affected Resource(s) and/or Data Source(s)

aws_msk_cluster

Potential Terraform Configuration

No response

References

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-msk-graviton3-m7g-instances-provisioned-clusters/

https://aws.amazon.com/blogs/big-data/amazon-msk-now-provides-up-to-29-more-throughput-and-up-to-24-lower-costs-with-aws-graviton3-support/

Would you like to implement a fix?

No

github-actions[bot] commented 9 months ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

aristosvo commented 9 months ago

Support is dependent on the update in the Go SDK for AWS and the AWS APIs themselves, which seem now the limiting factor. The error shown is not thrown by the aws_msk_cluster itself.

As soon as these are updated, it will work instantly.

alexanderscott commented 8 months ago

AWS unfortunately does not allow updating the ec2 instance family for MSK brokers in-place from m5 to m7g. This restriction is evident in the console as well. But I was just able to provision a new MSK cluster using m7g graviton brokers without issue. So I think this issue can be resolved, even though the error messaging is misleading.

kappa8219 commented 8 months ago

Interesting question will AWS provide support of in-place upgrade, cause price for Gravitons is almost equal to M-s. It seems to be the motivation for switchover. Mabe replication can help to perform upgrade without losing data but it requires iam, not sure all kafka clients (with old versions) are iam auth capable.

ajithshetty commented 7 months ago

Are there any order to follow to be able to launch m7 series. I had T series, which I destroyed and tried running with new m7 which did not work. Tried launching with new state location, which did not work either. Surprisingly, launching in new region freshly, works.

micr01996 commented 7 months ago

Is there any solution on this, what would be right approach here to switch to m7 series from m5?

Denton-L commented 6 months ago

We emailed AWS a month ago asking if it's possible to do an in-place upgrade from m5 to m7g instances in MSK and we got the following reply:

Hello there,

Greetings from AWS! Smitha ere from AWS support. Trsut this email finds you well.

I understand that you have an MSK cluster that uses M5 instances, and you would like to migrate to the M7g instance type. However, you are currently unable to do so and would like to know if migration to Graviton instances is currently supported. Let me look into this for you and assist you further.

Upon investigating this issue both internally and through the AWS documentation, I am pleased to inform you that AWS Graviton3-based M7g instances are now generally available for use with newly provisioned Amazon MSK clusters. These instances offer significant improvements, including up to a 24% price/performance boost, up to 29% higher write and read throughput, and up to a 27% reduction in CPU usage when compared to MSK clusters running on M5 instances.

However, it's important to note that Amazon MSK currently does not support upgrading an existing cluster from an "M5" instance type to an "M7g" instance type. Instead, the recommended approach is to create a new cluster directly with the desired "M7g" instance type.

For detailed information regarding Graviton3-based M7g instances, please refer to the following documentation: [+] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-msk-graviton3-m7g-instances-provisioned-clusters/

The reason for this limitation lies in the fact that brokers can only be upgraded within the same family type. The M7g instance is a Graviton-based type, whereas the M5.4xlarge is not. Consequently, to leverage the advantages of M7g Graviton instances, it is necessary to create a new cluster provisioned cluster ans select M7g as your instance type of choice.

Having said all the above, If the option to upgrade the M5 instances to M7g instance is available, Customers will be notified through our website: https://aws.amazon.com/new/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&whats-new-content-all.sort-order=desc&awsf.whats-new-categories=*all

I apologize for any inconvenience this limitation may cause, and I appreciate your understanding. Should you have any further questions or concerns regarding this, please feel free to reach out. I am here to assist you.

Thank you for choosing AWS, and I wish you a great day ahead!

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards, Smitha Amazon Web Services

edvardfagerholmsc commented 5 months ago

Incredible how much fluff they managed to write into an e-mail in order to just say that "In-place upgrading from x86 to Graviton is currently not supported."

banthaherder commented 3 months ago

Support is dependent on the update in the Go SDK for AWS and the AWS APIs themselves, which seem now the limiting factor. The error shown is not thrown by the aws_msk_cluster itself.

As soon as these are updated, it will work instantly.

Bump on updating to enable support for graviton instances in newly created MSK clusters 👍

hashicorp / terraform-provider-aws