Ftr : want to parse the content of each Kafka protocol package

[pantianying]

Hello, I am using this library as a proxy for our Kafka cluster. Now we have a requirement to change some data of Kafka packets, such as topic content, at the Kafka proxy level. So we need to parse each package, rather than simply do 4-tier proxy. I hope we can discuss this . thank you very mush~

[everesio]

Hi,

I am happy that you find the tool useful ;-)

Kafka-proxy modifies only host and port fields in Metadata and FindCoordinator responses. Other messages are simply passed through avoiding any unnecessary memory allocations.

It was not a goal of this project to be part of a library etc. Main goals were:

1. access kafka running in kubernetes
2. allow to access kafka through sock5 and http proxies
3. allow local sasl termination
4. provide sasl initilized by proxy
5. provide additional security not provided by Kafka

Coming back to your intention.

Currently there are 48 message types * 2 for request and response. Each message can have multiple versions e.g. Metadata has currently 10 different versions. Additionally there are 3 request and response header versions which are coupled to message versions.

If you want to be able to modify any message, you must be able to deserialize and serialize any message version ( which of course will affect performance)

If I were you I would start with some generator, taking some textual input and generating decoders and encoders. Kafka itself generates such java codecs for its internal usage. You should also expect that any new Kafka version can provide new message types or message versions.

Kind regards, Michal

[pantianying]

Hi,

I am happy that you find the tool useful ;-)

Kafka-proxy modifies only host and port fields in Metadata and FindCoordinator responses. Other messages are simply passed through avoiding any unnecessary memory allocations.

It was not a goal of this project to be part of a library etc. Main goals were:

1. access kafka running in kubernetes
2. allow to access kafka through sock5 and http proxies
3. allow local sasl termination
4. provide sasl initilized by proxy
5. provide additional security not provided by Kafka

Coming back to your intention.

Currently there are 48 message types * 2 for request and response. Each message can have multiple versions e.g. Metadata has currently 10 different versions. Additionally there are 3 request and response header versions which are coupled to message versions.

If you want to be able to modify any message, you must be able to deserialize and serialize any message version ( which of course will affect performance)

If I were you I would start with some generator, taking some textual input and generating decoders and encoders. Kafka itself generates such java codecs for its internal usage. You should also expect that any new Kafka version can provide new message types or message versions.

Kind regards, Michal

Thank you so detailed reply, I encountered some problems in the use, can you give me a contact information. I also want to meet a German friend. My email is pantianying@gmail.com

[pantianying]

In use, I found that Kafka proxy can only be used when it is deployed with the client side, but not independently or on the Kafka server-side machine. What is the principle of this. the client print 'panic: dial tcp 127.0.0.1:12851: connect: connection refused'

[everesio]

@pantianying I am not aware of any restriction why kafka-proxy cannot be used on the server side. Your problem is probably attributed to some misconfiguration.

BTW. If you want to send my a private message you can use an email from the release commits.

[bestlove4ling]

@pantianying modify code here