search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Advertised address for dynamic listeners #56

Closed Chrisss93 closed 4 years ago

Chrisss93 commented 4 years ago

If Kafka-proxy is not run as a local side-car but on a remote server to proxy a remote Kafka Cluster, the dynamic listeners for the discovered brokers from the metadata are not resolvable. We can side-step this issue by configuring a bootstrap-server-mapping for every kafka broker in the cluster we are proxying, but that feels a bit too manual and requires foreknowledge of how many brokers are in the cluster. Instead it seems simple enough to add another flag that sets the advertised address for any dynamic listeners.

Consider:

kafka-proxy server --dynamic-listeners-disable \
  --bootstrap-server-mapping remote-server:9092,0.0.0.0:9092,proxy-server:9092 \
  --bootstrap-server-mapping remote-broker1:9092,0.0.0.0:9093,proxy-server:9093 \
  --bootstrap-server-mapping remote-broker2:9092,0.0.0.0:9094,proxy-server:9094 \
  ...

vs.

kafka-proxy server --bootstrap-server-mapping remote-server:9092,0.0.0.0:9092,proxy-server:9092 \
  --default-listener-ip 0.0.0.0 \
  --dynamic-sequential-min-port 9093 \
  --dynamic-advertised-listener proxy-server

Let me know if this would be useful, or if I'm missing something on how kafka-proxy is intended to be used.

everesio commented 4 years ago

Hi @Chrisss93 , thank you for your contribution! I like the idea to have a possibility to configure dynamic listener advertise address.

I would rather take the value of default-listener-ip instead of 127.0.0.1 as default for dynamic-advertised-listener. This would keep current behavior. Additionally ListenDynamicInstance function must return advertised listener.

Please have a look at my changes ;-) https://github.com/grepplabs/kafka-proxy/commit/b6f326d673e18f9f03e840c8a20d16cdd5d95500