search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
474 stars 81 forks source link

What is the purpose of starting dynamic listeners #121

Open adiletkabylbekov opened 1 year ago

adiletkabylbekov commented 1 year ago

I'm trying to understand kafka-proxy usage. My brokers architecture is

172.28.132.4:9093 - kafka1
172.28.132.5:9093 - kafka2
172.28.132.6:9093 - kafka3

3 broker servers with TLS enabled

my kafka-proxy config is

      --bootstrap-server-mapping=172.28.132.4:9093,127.0.0.1:9091
      --bootstrap-server-mapping=172.28.132.5:9093,127.0.0.1:9092
      --bootstrap-server-mapping=172.28.132.6:9093,127.0.0.1:9093
      --tls-enable
      --tls-ca-chain-cert-file=ca.crt
      --tls-client-cert-file=cert.pem
      --tls-client-key-file=cert.key.pem
      --tls-insecure-skip-verify

In startup logs i identified of starting additional ports and addresses (Dynamic listeners ):

time="2022-12-26T11:45:30+03:00" level=info msg="Starting kafka-proxy version 0.3.3"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.4:9093 advertised as 127.0.0.1:9091"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.5:9093 advertised as 127.0.0.1:9092"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.6:9093 advertised as 127.0.0.1:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9091 (127.0.0.1:9091) for remote 172.28.132.4:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9092 (127.0.0.1:9092) for remote 172.28.132.5:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9093 (127.0.0.1:9093) for remote 172.28.132.6:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Ready for new connections"
time="2022-12-26T11:45:30+03:00" level=info msg="New connection for 172.28.132.6:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka2.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:38469) for remote kafka2.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:38469 for broker kafka2.i:9093 advertised as 0.0.0.0:38469"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka1.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:33523) for remote kafka1.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:33523 for broker kafka1.i:9093 advertised as 0.0.0.0:33523"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka3.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:38675) for remote kafka3.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:38675 for broker kafka3.i:9093 advertised as 0.0.0.0:38675"
time="2022-12-26T11:45:30+03:00" level=info msg="New connection for 172.28.132.6:9093"
time="2022-12-26T11:46:00+03:00" level=info msg="New connection for kafka3.i:9093"
time="2022-12-26T11:53:12+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T11:56:00+03:00" level=info msg="Server 172.28.132.6:9093 closed connection"
time="2022-12-26T11:56:25+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T12:03:12+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T12:06:25+03:00" level=info msg="Server 172.28.132.6:9093 closed connection"
time="2022-12-26T12:35:05+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T12:45:05+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T12:51:37+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T13:02:00+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T15:55:38+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T16:05:39+03:00" level=info msg="Server kafka1.i:9093 closed connection"

I've used endpoints "bootstrap.servers": "127.0.0.1:9091,127.0.0.1:9092,127.0.0.1:9093" on my clients, without providing ssl certs, and when producers is pushing messages i've got errors

Could not publish message (error: KafkaError{code=UNKNOWN,val=-1,str="Unknown broker error"})

There are no any errors in brokers logs, and i'm suggesting that my kafka-proxy or client configurations is not correct.

So my questions are: Which endpoint i need to use in my clients? What is the reason of starting Dynamic listener? Is kafka-proxy terminating ssl on their own side and for karka-proxy clients connection is plaintext?

ronaldpetty commented 1 year ago

@adiletkabylbekov I am new to the proxy. Just curious, can you share one of your broker configs? I am guessing your listeners and advertised listeners differ (e.g. some reverse DNS lookup, dual stack, etc.). Not saying thats wrong, just thinking it might be the trigger.