MSK Connect support

[AlexanderSchiff]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

MSK Connect is MSK's equivalent of Confluent Connect. It allows loading of Kafka connectors.

New or Affected Resource(s)

• aws_msk_plugin
• aws_msk_worker
• aws_msk_connector

Potential Terraform Configuration

resource "aws_msk_plugin" "example" {
  name = "name"
  content_type = "JAR" # "JAR" or "ZIP"
  description = "desc" # optional
  location {
    s3_location {
      bucket_arn = aws_s3_bucket.bucket1.arn
      file_key = aws_s3_bucket_object.object1.id
      object_version = aws_s3_bucket_object.object1.version_id # optional
    }
  }
}

resource "aws_msk_worker" "example" {
  name = "name"
  description = "desc" # optional
  properties_file_content = filebase64("${path.module}/connect-distributed.properties")
}

resource "aws_msk_connector" "example" {
  capacity {
    auto_scaling {
      max_worker_count = 1
      mcu_count = 1
      min_worker_count = 1
      scale_in_policy {
        cpu_utilization_percentage = 1
      }
      scale_out_policy {
        cpu_utilization_percentage = 1
      }
    } # optional
    provisioned_capacity {
      mcu_count = 1
      worker_count = 1
    } # optional
  } # optional
  connector_configuration = {
    key1 = "value1"
    key2 = "value2"
  }
  connector_description = "desc" # optional
  connector_name = "name"
  kafka_cluster {
    apache_kafka_cluster {
      bootstrap_servers = aws_msk_cluster.example.bootstrap_brokers_sasl_iam
      vpc {
        security_groups = [aws_security_group.sg.id]
        subnets = [
          aws_subnet.subnet_az1.id,
          aws_subnet.subnet_az2.id,
          aws_subnet.subnet_az3.id,
        ]
      }
    }
  }
  kafka_cluster_client_authentication {
    authentication_type = "IAM" # IAM or NONE
  }
  kafka_cluster_encryption_in_transit {
    encryption_type = "TLS" # TLS or PLAINTEXT
  }
  kafka_connect_version = "1.0"
  log_delivery {
    worker_log_delivery {
      cloud_watch_logs {
        enabled = true
        log_group = "loggroupname"
      }
      firehose {
        enabled = true
        delivery_stream = "streamname"
      }
      s3 {
        bucket = aws_s3_bucket.bucket2.id
        enabled = true
        prefix = "pre"
      }
    }
  } # optional
  plugins = [
    {
      custom_plugin_arn = aws_msk_plugin.example.arn
      revision = aws_msk_plugin.example.revision
    }
  ]
  service_execution_role_arn = aws_iam_role.example.arn
  worker_configuration {
    revision = aws_msk_worker.example.revision
    worker_configuration_arn = aws_msk_worker.example.arn
  } # optional
}

References

• https://aws.amazon.com/about-aws/whats-new/2021/09/amazon-msk-connect-run-serverless-scalable-kafka-connect-clusters-amazon-msk/
• https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html
• https://docs.aws.amazon.com/MCK/latest/mck/mkc-api.pdf#Welcome

[ewbankkit]

@AlexanderSchiff Thanks for raising this issue. It has already been noticed in #20942. I'm going to close this one as a duplicate so that we can concentrate discussion in the linked issue. Please add any additional comments there.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.