search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
468 stars 81 forks source link

AWS MSK Serverless Connectivity Test Drops #134

Open kokokruunch opened 1 year ago

kokokruunch commented 1 year ago

Hi there, I had a question. I was able to set-up an MSK kafka proxy using what was provided:

kafka-proxy server --bootstrap-server-mapping "b473-xx.xx.amazonaws.com:9098,0.0.0.0:30001" \
                   --tls-enable --tls-insecure-skip-verify \
                   --sasl-enable \
                   --sasl-method "AWS_MSK_IAM" \
                   --sasl-aws-region "eu-central-1" \
                   --log-level debug

I then create an SSH tunnel (tunnelling through 30001) to allow my local machine to communicate to the proxy. When I test the connectivity (I'm using Conduktor), I'm seeing the following but at some point, it just gets disconnected. Am I missing something? Thanks.

INFO[2023-05-08T15:49:15+10:00] New connection for b473-xx.xx.amazonaws.com:9098
DEBU[2023-05-08T15:49:15+10:00] Successful IAM SASL handshake. Available mechanisms: [AWS_MSK_IAM]
DEBU[2023-05-08T15:49:15+10:00] Kafka request key 18, version 3, length 50
DEBU[2023-05-08T15:49:15+10:00] Kafka response key 18, version 3, length 253
DEBU[2023-05-08T15:49:15+10:00] Kafka request key 3, version 11, length 29
DEBU[2023-05-08T15:49:15+10:00] Kafka response key 3, version 11, length 896
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b437-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:41469) for remote b437-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:41469 for broker b437-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:41469
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b59-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:40465) for remote b59-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:40465 for broker b59-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:40465
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b104-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:37319) for remote b104-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:37319 for broker b104-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:37319
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b482-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:36847) for remote b482-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:36847 for broker b482-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:36847
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b95-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:35943) for remote b95-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:35943 for broker b95-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:35943
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b464-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:32951) for remote b464-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:32951 for broker b464-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:32951
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b446-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:34611) for remote b446-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:34611 for broker b446-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:34611
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b86-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:36133) for remote b86-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:36133 for broker b86-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:36133
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b455-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:40509) for remote b455-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:40509 for broker b455-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:40509
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b77-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:34419) for remote b77-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:34419 for broker b77-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:34419
INFO[2023-05-08T15:49:15+10:00] Starting dynamic listener for broker b68-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Listening on 0.0.0.0:0 (0.0.0.0:33737) for remote b68-xx.xx.amazonaws.com:9098
INFO[2023-05-08T15:49:15+10:00] Dynamic listener 0.0.0.0:33737 for broker b68-xx.xx.amazonaws.com:9098 advertised as 0.0.0.0:33737
DEBU[2023-05-08T15:49:15+10:00] Address mappings broker=b473-xx.xx.amazonaws.com:9098, listener=0.0.0.0:30001, advertised=0.0.0.0:30001
INFO[2023-05-08T15:49:50+10:00] Client closed local connection on 127.0.0.1:30001 from 127.0.0.1:43414 (b473-xx.xx.amazonaws.com:9098)
INFO[2023-05-08T15:49:50+10:00] New connection for b473-xx.xx.amazonaws.com:9098
DEBU[2023-05-08T15:49:50+10:00] Successful IAM SASL handshake. Available mechanisms: [AWS_MSK_IAM]
INFO[2023-05-08T15:49:50+10:00] Client closed local connection on 127.0.0.1:30001 from 127.0.0.1:45368 (b473-xx.xx.amazonaws.com:9098)