awslabs / data-solutions-framework-on-aws

An open-source framework that simplifies implementation of data solutions.
https://awslabs.github.io/data-solutions-framework-on-aws/
Apache License 2.0
105 stars 14 forks source link

Add support to MSK KRaft #664

Closed armaseg closed 3 weeks ago

armaseg commented 3 weeks ago

AWS launched support for KRaft on new clusters on Amazon MSK starting from version 3.7. https://aws.amazon.com/blogs/big-data/introducing-support-for-apache-kafka-on-raft-mode-kraft-with-amazon-msk-clusters/

Currently, MSK Provisioned cluster depends on zookeeper, so if you try to deploy a new cluster using KRaft, you will get the following error:

Ex:

version = dsf.streaming.KafkaVersion.of('3.7.x.kraft')

msk_cluster = dsf.streaming.MskProvisioned(self, "cluster",
            vpc=vpc,
            cluster_name="my-cluster-3-6-0",
            kafka_version=version,
            subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT),
            security_groups=[ec2.SecurityGroup.from_security_group_id(
                self,
                id='sg',
                security_group_id="sg-1234"
            )],
            client_authentication=dsf.streaming.ClientAuthentication.sasl(
                iam=True
            )
        )

Returns the following error:

Received response status [FAILED] from custom resource. Message returned: Error: Cannot read properties of undefined (reading 'split') TypeError: Cannot read properties of undefined (reading 'split'), at Runtime.onEventHandler [as handler] (file:///var/task/index.mjs:23:76), at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at P (/var/task/index.js:1:1756) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Runtime.handler (/var/task/__entrypoint__.js:1:932)

In the future, the Apache Kafka community plans to remove the ZooKeeper mode entirely.

lmouhib commented 3 weeks ago

Already tracked in #640. Closing this one as duplicate.