Provider wants to replace existing kafka_acls when a new entry/ creating new resource is being added.

[thennati]

Hi @Mongey ,

thank you for amazing tool. we're seeing an issue when we try to create a new resource, or update the resource its trying to destroying existing acls and creating new acls. is there any fix you can provide us. Really appreciate your help on this. _im using 0.6.0 version, using Amazon Managed Kafka. running terraform in an EC2 Terraform v1.5.7 on linuxamd64 output: build 11-Apr-2024 09:45:06  # kafka_acl.acl[61] will be created build 11-Apr-2024 09:45:06  + resource "kafka_acl" "acl" { build 11-Apr-2024 09:45:06 + acl_host = "" build 11-Apr-2024 09:45:06 + acl_operation = "Read" build 11-Apr-2024 09:45:06 + acl_permission_type = "Allow" build 11-Apr-2024 09:45:06 + acl_principal = "User:CN=test.user" build 11-Apr-2024 09:45:06 + id = (known after apply) build 11-Apr-2024 09:45:06 + resource_name = "test_dev" build 11-Apr-2024 09:45:06 + resource_pattern_type_filter = "Literal" build 11-Apr-2024 09:45:06 + resource_type = "Group" build 11-Apr-2024 09:45:06 } build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06  # kafka_acl.acl[62] will be created build 11-Apr-2024 09:45:06  + resource "kafka_acl" "acl" { build 11-Apr-2024 09:45:06 + acl_host = "" build 11-Apr-2024 09:45:06 + acl_operation = "Read" build 11-Apr-2024 09:45:06 + acl_permission_type = "Allow" build 11-Apr-2024 09:45:06 + acl_principal = "User:CN=test.user" build 11-Apr-2024 09:45:06 + id = (known after apply) build 11-Apr-2024 09:45:06 + resource_name = "kafka_topic" build 11-Apr-2024 09:45:06 + resource_pattern_type_filter = "Prefixed" build 11-Apr-2024 09:45:06 + resource_type = "Topic" build 11-Apr-2024 09:45:06 } build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06 Plan: 58 to add, 0 to change, 55 to destroy.

few open and closed issues https://github.com/Mongey/terraform-provider-kafka/issues/285 https://github.com/Mongey/terraform-provider-kafka/issues/55#issuecomment-601356292

[Mongey]

Hi @thennati can you provide the full terraform you are using, and what version of Kafka this is on.

Thanks!

[thennati]

MSK kafka version : 3.5.1

provider.tf :

terraform {
  required_version = ">=1.1.5"
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}
provider "kafka" {
  bootstrap_servers = [var.bootstrap_servers]
  ca_cert           = file("${path.cwd}/../certs/ca-chain.crt")
  client_cert       = file("${path.cwd}/../certs/msk-acls.pem")
  client_key        = file("${path.cwd}/../certs/private.key")
  tls_enabled       = true
}

terraform {
  backend "s3" {
  }
}

main.tf :

locals {
  acl_pairs = flatten([
    for acl_pair in var.kafka_acl : [
      for name in acl_pair.resource_name : {
        resource_name = name
        resource_type = acl_pair.resource_type
        acl_principal = acl_pair.acl_principal
        acl_operation = acl_pair.acl_operation
        acl_permission_type = acl_pair.acl_permission_type
        resource_pattern_type_filter = acl_pair.resource_pattern_type_filter
      }
    ]
  ])
}
resource "kafka_acl" "acl" {
    count = length(local.acl_pairs)
    resource_name       = local.acl_pairs[count.index].resource_name
    resource_type       = local.acl_pairs[count.index].resource_type
    acl_principal       = local.acl_pairs[count.index].acl_principal
    acl_host            = "*"
    acl_operation       = local.acl_pairs[count.index].acl_operation
    acl_permission_type = local.acl_pairs[count.index].acl_permission_type
    resource_pattern_type_filter = local.acl_pairs[count.index].resource_pattern_type_filter
}

variables.tf :

variable "kafka_acl" {
  type = map(object({
    resource_name       = list(string)
    resource_type       = string
    acl_principal       = string
    acl_operation       = string
    acl_permission_type = string
    resource_pattern_type_filter = string
  }))
  default = {
    "Consumer_1" = {
      resource_name       = ["syslog"]
      resource_type       = "Topic"
      acl_principal       = "User:Alice"
      acl_operation       = "Write"
      acl_permission_type = "Deny"
      resource_pattern_type_filter = "Literal"
    }
    "Consumer_2" = {
      resource_name       = ["syslog2"]
      resource_type       = "Topic"
      acl_principal       = "User:Alice"
      acl_operation       = "Read"
      acl_permission_type = "Deny"
      resource_pattern_type_filter = "Prefixed"
    }
  }

}
variable "bootstrap_servers" {
  type    = string
  default = "broker.kafka.ap-southeast-2.amazonaws.com:9094"
}

variable "ca-chain"{
  type = string
  default = "../certs/ca-chain.crt"
}

variable "cert"{
  type = string
  default = "../certs/msk-acls.pem"
}

variable "private_key" {
  type = string
  default = "../certs/private.key"
}

terrform.acls.tfvars :

  "kafka_ui_topic"= {
    resource_name       = ["*"]
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user_2"
    acl_operation       = "All"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }
  "relay_acl_write"= {
    resource_name       = ["*"] #relay to access all the topics avaialble in the cluster
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user_1" 
    acl_operation       = "Write"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }
  "relay_acl_read"= {
    resource_name       = ["*"] #relay to access all the topics avaialble in the cluster
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user"
    acl_operation       = "Read"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }

[thennati]

please let us know if you still require any further info on this thanks @Mongey

[thennati]

@Mongey any update on this please ?

[Mongey]

@thennati I haven't had time to look at this yet

[thennati]

@Mongey can you please have a look at this reply from sarama https://github.com/IBM/sarama/issues/2885

[joaocc]

@Mongey could you please kindly confirm if this is planned to be addressed? If not, is the recommendation to use IAM (assuming it is fixed in 0.7.x)? Thanks