search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Proxy to multiple cluster bases on header fields #70

Closed ut0mt8 closed 4 years ago

ut0mt8 commented 4 years ago

I known it's not the goal of this project but I have in mind something that can help me to solve big scaling issues. The idea is to write a proxy between the producer and the brokers cluster, that can route all the trafic to the appropriate kafka cluster depending on a field on kafka header message. Maybe my idea is just crap, I don't know but if I can your input I ll appreciate.

everesio commented 4 years ago

Apart from Produce message which contains message headers, a kafka client always sends ApiVersions and Metadata requests. Theses manages are required by kafka protocol to set up a connection to the broker(s).

A proxy / dispatcher you propose would have to manage connections and topology changes to all possible brokers from all clusters. Some messages (apart from produce) would have to be replicated.

The nature of the native kafka protocol would make development very complex.

Better solution would be to build a proxy like gprc (with streaming) to multiple Kafkas. Of course the clients would have to use different protocol.

ut0mt8 commented 4 years ago

Thks for the answer. After reviewing the kafka protocol that what I though also.