search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 87 forks source link

Trying to proxifying Kafka from internal network to Aiven Kafka cluster #88

Closed remynollet closed 3 years ago

remynollet commented 3 years ago

Hello,

I have a Aiven Cluster and I have install a kafka-proxy on a kubernetes pod to allow some clients to access to the brokers via the internal network.

The pod is deployed with this configuration: 

   spec:
      containers:
      - name: vcstream-uat-proxy
        image: grepplabs/kafka-proxy:latest
        command:
        - '/bin/sh'
        - '-c'
        - |
          echo $ca_cert | base64 -d  > ca_cert.pem && \
          echo $access_cert | base64 -d  > access.cert && \
          echo $access_key | base64 -d  > access.key && \
          /opt/kafka-proxy/bin/kafka-proxy server \
          --log-format=json \
          --log-level=debug \
          --debug-enable \
          --bootstrap-server-mapping=10.42.64.33:12658,0.0.0.0:12658,10.48.18.69:12658 \
          --bootstrap-server-mapping=10.42.64.34:12658,0.0.0.0:12659,10.48.18.69:12659 \
          --bootstrap-server-mapping=10.42.64.35:12658,0.0.0.0:12660,10.48.18.69:12660 \
          --dynamic-advertised-listener 10.48.29.37 \
          --tls-enable \
          --tls-ca-chain-cert-file ca_cert.pem \
          --tls-client-cert-file access.cert \
          --tls-client-key-file access.key \
          --tls-client-key-password $keystore_password \
          --tls-insecure-skip-verify

Where 10.48.29.37 is the internal ip of the TCP loadbalancer.

When I try to kafkacat -b 10.48.29.37:12658 -L, it works fine. But when I try to consume or product any message, I have an issue.

%7|1631003957.028|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003957.028|STATE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Broker changed state INIT -> TRY_CONNECT
%7|1631003957.028|CONNECT|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: broker in state TRY_CONNECT connecting
%7|1631003957.028|STATE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Broker changed state TRY_CONNECT -> CONNECT
%7|1631003957.029|CONNECT|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Connecting to ipv4#10.48.18.69:12659 (plaintext) with socket 14
%7|1631003957.029|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003957.029|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003957.029|CONNECT|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Connected to ipv4#10.48.18.69:12659
%7|1631003957.029|CONNECTED|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Connected (#1)
%7|1631003957.030|FEATURE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1631003957.030|STATE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1631003957.030|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003958.031|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003959.032|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003960.033|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003961.034|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003962.036|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003963.037|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003964.038|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003965.040|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%7|1631003966.041|TOPPAR|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: private_dkt_out_listener_kafka_geco_v1 [4] 1 message(s) queued but broker not up
%5|1631003967.042|REQTMOUT|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Timed out ApiVersionRequest in flight (after 10012ms, timeout #0)
%7|1631003967.042|FAIL|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (see api.version.request configuration) (after 10012ms in state APIVERSION_QUERY) (_TRANSPORT)
%4|1631003967.042|FAIL|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (see api.version.request configuration) (after 10012ms in state APIVERSION_QUERY)
%7|1631003967.042|FEATURE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Updated enabled protocol features -ApiVersion to
%7|1631003967.043|STATE|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Broker changed state APIVERSION_QUERY -> DOWN
%4|1631003967.043|REQTMOUT|rdkafka#producer-1| [thrd:10.48.18.69:12659/16]: 10.48.18.69:12659/16: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests

Do you have any idea that could help us ?

everesio commented 3 years ago

Could you provider the configuration of the brokers and clients ?
Which broker version are you trying to connect to ?

remynollet commented 3 years ago

OK we have find the resolution of this issue.

In the --bootstrap-server-mapping we had set the IP of the Kafka-proxy service as the advertising server instead of setting the IP of the loadbalancer in front of service.

Now it works fine. The next step is to manage the "multi-instance" of the proxy and the "TLS Authentication".