Open JohnDoePBabu opened 5 years ago
This is probably miss configuration. How are your kafka brokers and kafka clients configured, which versions are used ? How do you start kafka-proxy ?
I am getting this as well. My topology is VPC peering my AWS to confluent.cloud and trying to get my localhost (outside of AWS env) to be able to reach confluent.cloud.
Could you provide server, proxy and client configuration ?
This is the environment I have setup:
https://docs.confluent.io/current/cloud/vpc.html
I don't have many settings on the server side available to me but I do know that I can connect to it on server:9092
from an instance running on my AWS env.
The proxy settings I am currently trying are:
kafka-proxy server --bootstrap-server-mapping "server:9092,0.0.0.0:9092" \
--log-level debug \
--tls-enable \
--tls-insecure-skip-verify
The client settings I am using are something to the affect of:
// kafka client version 2.4.0
import org.apache.kafka.clients.admin.{AdminClient, AdminClientConfig}
import java.util.Properties
val config = ConfigFor("environment")
val properties = new Properties()
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, config.bootstrapServerUrl)
properties.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "300000")
properties.put("ssl.endpoint.identification.algorithm", "https")
properties.put("sasl.mechanism", "PLAIN")
properties.put("sasl.jaas.config", s"""org.apache.kafka.common.security.plain.PlainLoginModule required username="${config.clusterAPIKey}" password="${config.clusterAPISecret}";""")
properties.put("security.protocol", "SASL_SSL")
AdminClient.create(properties)
I did try with jaas
on the proxy side with the same credentials, but was receiving the same error.
My specific error looks like:
INFO[2020-02-04T12:10:10Z] Starting kafka-proxy version 0.1.6
INFO[2020-02-04T12:10:10Z] Bootstrap server server:9092 advertised as 0.0.0.0:9092
INFO[2020-02-04T12:10:10Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote server:9092
INFO[2020-02-04T12:10:10Z] Ready for new connections
INFO[2020-02-04T12:11:31Z] New connection for server:9092
DEBU[2020-02-04T12:11:31Z] Sending SaslHandshakeRequest mechanism: PLAIN version: 0
DEBU[2020-02-04T12:11:31Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER]
DEBU[2020-02-04T12:11:31Z] Sending authentication opaque packets, mechanism PLAIN
DEBU[2020-02-04T12:11:31Z] Kafka request key -18175, version 0, length 369296128
INFO[2020-02-04T12:11:31Z] Reading data from local connection on 172.28.196.187:9092 from 172.20.80.195:60232 (server:9092) had error: api key -18175 is invalid
Where 172.28.196.187
is the kafka-proxy
IP and 172.20.80.195
is my localhost.
LMK if there is anything else I can provide and I am digging into our env more to see what I can find out. Thanks!
if it is not, use the correct TLS port or do not use "--tls-enable" in the proxy. If you want terminate TLS on proxy please use "--proxy-listener-tls-enable"
@hekaldama Did you get kafka-proxy working properly with socks or http proxying towards Confluent Cloud? Very interested in any working configs if that is the case. Cheers.
@datafisk I was not able to work on it anymore after my original comment. I will definitely post back here if I figure anything out. We are going to be setting up AWS transit gateway to at least allow VPN -> confluent.cloud access.
@hekaldama Did you get kafka-proxy working properly with socks or http proxying towards Confluent Cloud? Very interested in any working configs if that is the case. Cheers.
I can forward-proxy to Confluent Cloud via squid but not via sasl_ssl. Only when I set sasl_plaintext on the client side.
I think ii got this sorted @sarwarbhuiyan , i will DM you a link.
I am running in to similar issues when connecting to confluent cloud from the kafka-proxy and wanted to see if anyone was successful with that:
had error: api key XXXX is invalid
My server command:
./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \
--debug-enable \
--log-level="debug" \
--tls-enable \
--tls-insecure-skip-verify
And the kafka properties on the server:
bootstrap.servers=confluent-server:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ API_KEY }}" password="{{ API_SECRET }}";
Does anything jump out from my configuration or kafka-proxy setup?
@afoley-st don't you miss --sasl-enable
option?
@afoley-st don't you miss
--sasl-enable
option?
The same issue arises:
./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \
--debug-enable \
--log-level="debug" \
--sasl-enable \
--sasl-username="${SASL_USERNAME}" \
--sasl-password="${SASL_PASSWORD}" \
--tls-enable --tls-insecure-skip-verify
INFO[2021-06-02T13:22:40Z] Starting kafka-proxy version 0.2.8
INFO[2021-06-02T13:22:40Z] Bootstrap server confluent-server:9092 advertised as 0.0.0.0:9092
INFO[2021-06-02T13:22:40Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote confluent-server:9092
INFO[2021-06-02T13:22:40Z] Ready for new connections
INFO[2021-06-02T13:22:50Z] New connection for confluent-server:9092
DEBU[2021-06-02T13:22:50Z] Sending SaslHandshakeRequest mechanism: PLAIN version: 0
DEBU[2021-06-02T13:22:50Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER]
DEBU[2021-06-02T13:22:50Z] Sending authentication opaque packets, mechanism PLAIN
DEBU[2021-06-02T13:22:50Z] Kafka request key -3327, version 1, length 369296129
INFO[2021-06-02T13:22:50Z] Reading data from local connection on XXXXX:9092 from XXXXX:61791 (confluent-server:9092) had error: api key -3327 is invalid
Is the --proxy-listener-tls-enable
setting a requirement?
@afoley-st no --proxy-listener-tls-enable
should not be requirement. What version of client/server you have? What is your listener.security.protocol.map
@afoley-st no
--proxy-listener-tls-enable
should not be requirement. What version of client/server you have? What is yourlistener.security.protocol.map
It's running Apache Kafka Version 2.7.0
. I can't get the entire listener.security.protocol.map
(don't have access because its a managed service via confluent), but the bootstrap endpoint is SASL_SSL://confluent-server:9092
and client kafka version?
and client kafka version?
I've tried with both 2.7.0
and 2.8.0
@afoley-st did you try also with 2.3 see https://github.com/grepplabs/kafka-proxy/issues/28#issuecomment-582160739
@afoley-st skip the properties for your kafka client, the kafka-proxy has the required settings as per this config and your client will just initiate the connection to the proxy with just the bootstrap as required config, the proxy takes care of the rest.
@afoley-st don't you miss
--sasl-enable
option?The same issue arises:
./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \ --debug-enable \ --log-level="debug" \ --sasl-enable \ --sasl-username="${SASL_USERNAME}" \ --sasl-password="${SASL_PASSWORD}" \ --tls-enable --tls-insecure-skip-verify INFO[2021-06-02T13:22:40Z] Starting kafka-proxy version 0.2.8 INFO[2021-06-02T13:22:40Z] Bootstrap server confluent-server:9092 advertised as 0.0.0.0:9092 INFO[2021-06-02T13:22:40Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote confluent-server:9092 INFO[2021-06-02T13:22:40Z] Ready for new connections INFO[2021-06-02T13:22:50Z] New connection for confluent-server:9092 DEBU[2021-06-02T13:22:50Z] Sending SaslHandshakeRequest mechanism: PLAIN version: 0 DEBU[2021-06-02T13:22:50Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER] DEBU[2021-06-02T13:22:50Z] Sending authentication opaque packets, mechanism PLAIN DEBU[2021-06-02T13:22:50Z] Kafka request key -3327, version 1, length 369296129 INFO[2021-06-02T13:22:50Z] Reading data from local connection on XXXXX:9092 from XXXXX:61791 (confluent-server:9092) had error: api key -3327 is invalid
Is the
--proxy-listener-tls-enable
setting a requirement?
@afoley-st you should connect to your kafka-proxy server without SASL credentials/SSL (if are doing it right now, kafka-proxy already authenticates/encrypts connection with confluent kafka) as @datafisk mentioned
If I enable SASL, it's working perfectly but, If I remove --sasl-enable
flag and try to connect from client, this error will appear.
I was following this guide to connect to cluster. https://docs.conduktor.io/kafka-cluster-connection/setting-up-a-connection-to-kafka/connecting-to-a-secure-kafka
If I enable SASL, it's working perfectly but, If I remove
--sasl-enable
flag and try to connect from client, this error will appear.I was following this guide to connect to cluster. https://docs.conduktor.io/kafka-cluster-connection/setting-up-a-connection-to-kafka/connecting-to-a-secure-kafka
After few hours debug, I realized the problem is protocol I’m using. Just changed it to SASL_PLAINTEXT
and then everything works perfectly.
Updating this ticket for the SASL passthrough issue against Confluent Cloud, that generates the API key error. Since Confluent Cloud uses. SASL_PLAN over TLS the proxy needs to be setup with TLS listeners and it will translate/proxy/passthrough the connection details to Confluent Cloud. Below is an examle:
kafka-proxy server \
--bootstrap-server-mapping \
"pkc-lgwgm.eastus2.azure.confluent.cloud:9092,localhost:9092" \
--log-level DEBUG \
--proxy-listener-cert-file ./proxy.cert \
--proxy-listener-tls-enable \
--proxy-listener-key-file ./proxy.key \
--tls-enable
Then you just setup the kafka client to talk to the proxy, localhost in my example, so from kafkacat:
export USERNAME=<API_KEY/SASL_USERNAME>
export PASSWORD=<API_SECRET_KEY/SASL_PASSWORD>
export BROKERS=localhost:9092
kafkacat -b $BROKERS -L -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN -X sasl.username=$USERNAME -X sasl.password=$PASSWORD
If you use self-signed certs for some testing you want to add -X enable.ssl.certificate.verification=false
to kafkacat to make testing the passthrough simpler.
can anyone post a fully working config + start scripts for: kafka-proxy -> squid -> confluent kafka ?
I'm not able to get this to work even without squid. I've tried a number of different CLI options to kafka-proxy and with confluent-7.3.0/bin/kafka-console-producer, none of it works for me.
I keep getting "had error: api key -10495 is invalid" when I try to proxy a Kafka cluster with SASL. Is this because the Kafka broker version is too old? I'm using SASL plain with TLS enabled and the handshake is successful. Could someone help?