Hey,
Currently we have a kafka-cluster with 4 brokers in AWS, production environment, that doesn't need any authentication. The consumers and producers are deployed into other aws accounts then the Kafka cluster. The new Kafka cluster is deployed under a different account with 6 brokers and it require SASL authentication. So the clients are running in Aws Account A, old kafka cluster is in Aws Account B, new cluster is in Aws Account C.
Each of the current brokers are listening on port 9093, but have different hostname. Under Aws Account B there is a VPC Endpoint Service that point to the Network Load Balancer of the specific broker. The new Kafka cluster's brokers are listenning on different port, and has the same hostname.
What is your recommendation for the configuration for this situation?
What I tried: Created 4 ECS Services and kafka-proxy is configured in this way:
Service1: Listening on 9093 port and forward traffic to new cluster's broker to 9093 port
Service2: Listening on 9093 port and forward traffic to new cluster's broker to 9094 port
Service3: Listening on 9093 port and forward traffic to new cluster's broker to 9095 port
Service4: Listening on 9093 port and forward traffic to new cluster's broker to 9096 port
it means 2 brokers of the new cluster are not configured in kafka-proxy.
If I configure the kafka-proxy to use the cluster's bootstrap broker that is listening on port 9092, I have to define all of the brokers too in the proxy (without it the service fails, as the broker lists are received from the kafka metadata and the consumer's cannot connect to)
VPC Endpoint Services are rerouted the new Services' NLB. It seems this solution is working properly, but I'm not sure if it's the most optimal one.
Hey, Currently we have a kafka-cluster with 4 brokers in AWS, production environment, that doesn't need any authentication. The consumers and producers are deployed into other aws accounts then the Kafka cluster. The new Kafka cluster is deployed under a different account with 6 brokers and it require SASL authentication. So the clients are running in Aws Account A, old kafka cluster is in Aws Account B, new cluster is in Aws Account C. Each of the current brokers are listening on port 9093, but have different hostname. Under Aws Account B there is a VPC Endpoint Service that point to the Network Load Balancer of the specific broker. The new Kafka cluster's brokers are listenning on different port, and has the same hostname. What is your recommendation for the configuration for this situation?
What I tried: Created 4 ECS Services and kafka-proxy is configured in this way:
Service1: Listening on 9093 port and forward traffic to new cluster's broker to 9093 port Service2: Listening on 9093 port and forward traffic to new cluster's broker to 9094 port Service3: Listening on 9093 port and forward traffic to new cluster's broker to 9095 port Service4: Listening on 9093 port and forward traffic to new cluster's broker to 9096 port
it means 2 brokers of the new cluster are not configured in kafka-proxy. If I configure the kafka-proxy to use the cluster's bootstrap broker that is listening on port 9092, I have to define all of the brokers too in the proxy (without it the service fails, as the broker lists are received from the kafka metadata and the consumer's cannot connect to)
VPC Endpoint Services are rerouted the new Services' NLB. It seems this solution is working properly, but I'm not sure if it's the most optimal one.