search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.86k stars 9.21k forks source link

[Bug]: Unable to update MSK cluster policy after enabling multi-VPC connectivity #35538

Open Czeffik opened 10 months ago

Czeffik commented 10 months ago

Terraform Core Version

1.7.0

AWS Provider Version

5.34.0

Affected Resource(s)

aws_msk_cluster_policy

Expected Behavior

After enabling multi-VPC connectivity for MSK cluster it should be possible to update aws_msk_cluster_policy.

Actual Behavior

When I create a MSK cluster policy using aws_msk_cluster_policy I am receiving:

Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:${account_id}:cluster/${cluster_name}/${cluster_id}): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: ${request_id}, BadRequestException: Current cluster policy version needed for Update

during terraform plan it looks like terraform is not aware of existing aws_msk_cluster_policy - it does not modify resource but adds new one.

Relevant Error/Panic Output Snippet

Basically, terraform is not aware that `aws_msk_cluster_policy` exist and is not pulling the current policy version prior to being updated.

Terraform Configuration Files

resource "aws_msk_cluster_policy" "example" {
  cluster_arn = var.msk_cluster_arn

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Sid    = "ExampleMskClusterPolicy"
      Effect = "Allow"
      Principal = {
        "AWS" = "arn"aws:iam::[account_id]:root"
      }
      Action = [
        "kafka:Describe*",
        "kafka:Get*",
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
      ]
      Resource = var.msk_cluster_arn
    }]
  })
}

Steps to Reproduce

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

This issue looks similar to this one: https://github.com/hashicorp/terraform-provider-aws/issues/33107

Would you like to implement a fix?

None

github-actions[bot] commented 10 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

justinretzolk commented 10 months ago

Potentially related #34419 (see this comment specifically)

vishwa-trulioo commented 5 months ago

Is there any update on this? I raised the same issue sometime back and they rolled out a fix. but, looks like it's back again. How can this happen? https://github.com/hashicorp/terraform-provider-aws/issues/33107

vishwa-trulioo commented 5 months ago

@ewbankkit Sometime back you fixed this same issue with https://github.com/hashicorp/terraform-provider-aws/issues/33107. But, It looks to have come back. I even used same provider version which originally worked (5.15). Gives me this error.

aws_msk_cluster_policy.default: Creating...
╷
│ Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: ab940298-8635-4e78-8ef1-da15b2b55052, BadRequestException: Current cluster policy version needed for Update
│ 
│   with aws_msk_cluster_policy.default,
│   on data.tf line 12, in resource "aws_msk_cluster_policy" "default":
│   12: resource "aws_msk_cluster_policy" "default" {

Here is the output from the TF Debug log enabled,

aws_msk_cluster_policy.default: Creating...
2024-06-14T18:33:47.874-0700 [INFO]  Starting apply for aws_msk_cluster_policy.default
2024-06-14T18:33:47.875-0700 [DEBUG] aws_msk_cluster_policy.default: applying the planned Create change
2024-06-14T18:33:53.612-0700 [ERROR] provider.terraform-provider-aws_v5.20.0_x5: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 tf_req_id=a33a88be-c1f6-af9a-85df-92061840931e tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/aws @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR diagnostic_summary="setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update" tf_resource_type=aws_msk_cluster_policy tf_rpc=ApplyResourceChange timestamp=2024-06-14T18:33:53.612-0700
2024-06-14T18:33:53.636-0700 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-06-14T18:33:53.636-0700 [ERROR] vertex "aws_msk_cluster_policy.default" error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update
╷
│ Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update
│ 
│   with aws_msk_cluster_policy.default,
│   on data.tf line 12, in resource "aws_msk_cluster_policy" "default":
│   12: resource "aws_msk_cluster_policy" "default" {
│ 
╵
2024-06-14T18:33:53.661-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-06-14T18:33:53.667-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.20.0/darwin_amd64/terraform-provider-aws_v5.20.0_x5 pid=6146
2024-06-14T18:33:53.667-0700 [DEBUG] provider: plugin exited

Are you able to see if you can help with this? Thank you in advance.

vishwa-trulioo commented 5 months ago

Here is another thing I discovered. This error only shows up when Updating the MSK cluster policy. i.e. If you are adding Cluster policy for the first time to an MSK cluster it won't give error.

ivanbishop commented 1 week ago

Does the module let you add cross account multi-vpc into MSK now ?