search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Multiple kafka-proxy servers to balance the traffic #82

Closed ericsun2 closed 3 years ago

ericsun2 commented 3 years ago

In a scenario that we have 12 brokers in the cluster, can we setup 3 kafka-proxy servers like below, so that each kafka-proxy will handle traffic for 4 brokers?

kafka-proxy_by_grepplabs

It's not quite clear to me from the README, maybe each kafka-proxy must have all 12 brokers mapped to 12 ports, and we have to put an ALB (in AWS) in front. Please advise.

everesio commented 3 years ago

Hi, you can configure such topology configuring proxy01, proxy02 and proxy03 using bootstrap-server-mapping and external-server-mapping

e.g.

proxy01:
kafka-proxy server --bootstrap-server-mapping "b1-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32500,proxy01:32500" \
                   --bootstrap-server-mapping "b2-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32501,proxy01:32501" \
                   --bootstrap-server-mapping "b3-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32502,proxy01:32502" \
                   --bootstrap-server-mapping "b4-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32503,proxy01:32503" \
                   --external-server-mapping  "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500" \
                   --external-server-mapping  "b6-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32501" \
                   --external-server-mapping  "b7-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32502" \
                   --external-server-mapping  "b8-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32503" \
                   --external-server-mapping  "b9-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32500" \
                   --external-server-mapping  "b10-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32501" \
                   --external-server-mapping  "b11-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32502" \
                   --external-server-mapping  "b12-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32503" \
                   --dynamic-listeners-disable

proxy02:
kafka-proxy server --external-server-mapping  "b1-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32500" \
                   --external-server-mapping  "b2-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32501" \
                   --external-server-mapping  "b3-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32502" \
                   --external-server-mapping  "b4-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy01:32503" \
                   --bootstrap-server-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32500,proxy02:32500" \
                   --bootstrap-server-mapping "b6-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32501,proxy02:32501" \
                   --bootstrap-server-mapping "b7-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32502,proxy02:32502" \
                   --bootstrap-server-mapping "b8-xyz.c1.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:32503,proxy02:32503" \
                   --external-server-mapping  "b9-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32500" \
                   --external-server-mapping  "b10-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32501" \
                   --external-server-mapping  "b11-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32502" \
                   --external-server-mapping  "b12-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy03:32503" \
                   --dynamic-listeners-disable

proxy03:
....
everesio commented 3 years ago

You can also put multiple proxies with the same configuration behind a load balancer e.g.

enjoyear commented 3 years ago

Hi @everesio , thank you for your reply! Why do we use external-server-mapping here --external-server-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500". Will it work if I use --dial-address-mapping "b5-xyz.c1.kafka.us-east-1.amazonaws.com:9094,proxy02:32500"?

everesio commented 3 years ago

bootstrap-server-mapping starts a TLS listener and changes/remaps the kafka broker addresses in the response. external-server-mapping only changes the kafka broker addresses