Support of SSL/Kerberos

thexixx commented 7 years ago

Hello!

Could someone tell me if this lib supports SSL/Kerberos? At the momoent I can't find any .Net Kafka Client which is supporting SSL and(or) Kerberos.

edenhill commented 7 years ago

Is that python librdkafka on Windows or Linux?

edenhill commented 7 years ago

Also, did you find the cause of the broker exception?

[2017-05-25 18:20:15,694] DEBUG Connection with workstation/10.6.XX.XX disconnected (org.apache.kafka.common.network.Selector:375)
javax.net.ssl.SSLHandshakeException: certificate verify message signature error

TheMidgardWatcher commented 7 years ago

Is that python librdkafka on Windows or Linux?

python librdkafka runs on Linux OS

kavyashivakumar commented 7 years ago

@TheMidgardWatcher. @edenhill is there any example for .net consumer/producer using SASL?

TheMidgardWatcher commented 7 years ago

@kavyashivakumar Sorry - we don't use SASL consumers.

TheMidgardWatcher commented 7 years ago

@edenhill Also, did you find the cause of the broker exception?

No, after long investigations I'm not able to determine the cause of it...

SStar1314 commented 7 years ago

@edenhill Hi, Author. I also hit this issue in my server environment. The first time of running telegraf(with kafka_consumer re-writted, using librdkafka for ssl connection) is successful, but after once re-start service, the handshake of ssl connection always failed. It reports:

%3|1497259160.209|FAIL|rdkafka#consumer-1| [thrd:ssl://158.85.44.247:9093/bootstrap]: ssl://158.85.44.247:9093/bootstrap: SSL handshake failed: s3_both.c:406: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)

I used:
librdkafka : master branch kafka : kafka_2.11-0.10.2.0.tgz telegraf : telegraf_1.1.0_amd64.deb openssl : OpenSSL 1.0.2g 1 Mar 2016

I check the source code, the error happens on rd_kafka_transport_ssl_handhsake function in rdkafka_transport.c file. When executing "SSL_do_handshake", it always return "unexpected message" and error-return-value 2, which means "SSL_ERROR_WANT_READ".

My openssl connection result is ok, but it indeed report unexpected message.

openssl s_client -connect *:9093 :

CONNECTED(00000003) verify return:1 140414289491608:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:s3_both.c:406:

Server public key is 1024 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : DHE-DSS-AES128-GCM-SHA256 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1497260475 Timeout : 300 (sec) Verify return code: 0 (ok)

I checked two clue below, maybe something wrong with ssl3's stranger behavior. https://stackoverflow.com/questions/28011581/websocket-ssl-handshake-failure https://www.openssl.org/docs/man1.1.0/ssl/SSL_get_error.html

Is it any method to allow us not use ssl3 connection ?

TheMidgardWatcher commented 7 years ago

Hi there!

@edenhill - We found a couple of environments where simple consumer example works (Confluent kafka platform is the same). And now we are investigating why, and what the difference between working and non-working workstations.

@SStar1314 have you tried to run it on a different machines and don't get Handshake error?

UPD (2017/6/14): @edenhill We've found that people who reported that the example works fine, just didn't add the OnError handler, and they simply didn't see errors in console, but errors was there.

TheMidgardWatcher commented 7 years ago

@SStar1314 Have you tried to update your kafka to 0.10.2.1 version?

SStar1314 commented 7 years ago

@TheMidgardWatcher I tried to run the command on another similar environment, the handshake error happens as above attached. And after two days struggle work, the issue disappear for no-reason. I didn't update kafka's version, I re-build librdkafka for many times to dump error process, but make no effect. Then by chance, I add dump error message on Telegraf's kafka-consumer plugin , I re-build Telegraf, and after I restart Telegraf, the issue disappear, not report handshake error anymore. And use openssl command to communicate directly also don't report error message. So, I got two environments, both got the ssl handshake error, one fixed through re-build telegraf, another is hold for more investigation. No more clue. That fix is quite stranger, make no sense, I tried to reboot machine several times but make no changes.

TheMidgardWatcher commented 7 years ago

@SStar1314 we're fighting with this issue since rdlkafka-dotnet with no result. But i found This Kafka issue KAFKA-4959 that might be a reason of ssl handshake errors. So now we are upgrading our environments to check if issue is gone on kafka 0.10.2.1. I'd recommend you to do same thing.

SStar1314 commented 7 years ago

@TheMidgardWatcher Thanks. I tried kafka 0.10.2.1 today, not fix for my environment, issue still exist. If you config kafka server.properties to set ssl.client.auth=none, the handshake error disappear. I am wondering if there is mis-understanding usage about this config.

TheMidgardWatcher commented 7 years ago

Unfortunately, for me 0.10.2.1 update had no success too...

htims1989 commented 7 years ago

Followed the docs, got the exact same problem as SStar1314; as soon as I set ssl.client.auth=required on the broker I get: ssl://kafka1.XXXXX.com:9093/bootstrap: SSL handshake failed

Happy to provide any info required, just let me know what :)

EDIT: so I got some certs from our in house CA instead of using self signed and this seems to have helped somewhat. I only intermittently get the handshake error from each of the brokers in my cluster but can still consume everything fine.

EDIT2: So if I send my test client direct to a single broker I get the handshake/shutdown errors for every other broker in the cluster. This seems to be the case regardless of which one I point it at.

htims1989 commented 7 years ago

Still battling with this, things I've tried:

a cluster on Ubuntu 16.04 LTS
a cluster Windows Server 2012
enabling only TLSv1, then TLS v1.1 and finally TLSv1.2
disabling hostname verification (ssl.endpoint.identification.algorithm)
explicitly setting ssl.secure.random.implementation to SHA1PRNG
re-created certs and triple checked the keystores & truststores

I can consume all records from all topics the majority of the time despite the errors but it does occasionally fail completely with "5/5 brokers down".

Has anyone got any further?

TheMidgardWatcher commented 7 years ago

@edenhill Could you comment this posts above? Seems like this issue is more global than only someone's local environment or configuration...

edenhill commented 7 years ago

Please try librdkafka v0.11.0-RC2 which has some SSL error propagation fixes

TheMidgardWatcher commented 7 years ago

To @edenhill , just checked and got a bunch of this:

Error: Local_Ssl ssl://broker1:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
Error: Local_Ssl ssl://broker2:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
Error: Local_Transport ssl://broker1:9093/bootstrap: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init: 
Error: Local_Ssl ssl://broker3:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
Error: Local_AllBrokersDown 3/3 brokers are down
Error: Local_Ssl ssl://broker2:9093/bootstrap: SSL handshake failed: SSL syscall error number: 5: No error
Error: Local_Ssl ssl://broker1:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
Error: Local_Ssl ssl://broker3:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
Error: Local_Transport ssl://broker2:9093/bootstrap: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init: 
Error: Local_AllBrokersDown 3/3 brokers are down

edenhill commented 7 years ago

When is this occuring? Directly after connect? At regular intervals (say.. the broker idle connection reaper time (10min default))? Or suddenly? Does it happen for all brokers simultaneously? Are there any hints in the broker logs? Are there any occassions where this does not occur?

htims1989 commented 7 years ago

@edenhill

It's on first connect and reconnect (I paused the debugger earlier and it had to re-establish a connection). Once connected and consuming/producing it doesn't seem to re-occur.
It can happen for all brokers simultaneously which results in the "AllBrokersDown X/X brokers down" but it does manage to connect on the next attempt usually.
Nothing shows up in the broker logs.
If I delete all zookeeper and kafka data so I effectively have a brand new cluster, I sometimes won't get it right away but it appears later.
It seems completely random, but setting the client config to a single broker (any in the cluster) will usually result in errors from all other brokers. Errors from the single broker are then rare.

edenhill commented 7 years ago

It would be great if you could find the most minimal test case to reproduce this, preferably a single broker on localhost or similar, with a trivial client application.

treziac commented 7 years ago

I managed to replicate on localhost, by using two brokers.

Everything run on windows10, using kafka 0.11.0.0 and librdkafka 0.11.0-RC2

Broker 0: PLAINTEXT://:9092,SSL://:9093 Broker 1: PLAINTEXT://:9095,SSL://:9094

SSL configuration done with https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka, using openssl version mentioned here: https://github.com/edenhill/librdkafka/blob/master/README.win32

server.properties :

broker.id=0
listeners=PLAINTEXT://:9092,SSL://:9093
ssl.keystore.location=D:/kafka/ssl/broker_localhost_server.keystore.jks
ssl.keystore.password=abcdefgh
ssl.keystore.type=JKS
ssl.key.password=abcdefgh
ssl.truststore.location=D:/kafka/ssl/broker_localhost_server.truststore.jks
ssl.truststore.password=abcdefgh
ssl.truststore.type=JKS
ssl.protocol = TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

server2.properties:

broker.id=1
listeners=PLAINTEXT://:9095,SSL://:9094
ssl.keystore.location=D:/kafka/ssl/broker_localhost2_server.keystore.jks
ssl.truststore.location=D:/kafka/ssl/broker_localhost2_server.truststore.jks
...

librdkafka config:


                { "bootstrap.servers", brokerList },
                { "security.protocol", "ssl" },
                { "ssl.ca.location", @"D:/kafka/ssl/ca-cert" },
                { "ssl.certificate.location", @"D:/kafka/ssl/client_local_client.pem" },
                { "debug" , "security" },
                { "ssl.key.location", @"D:/kafka/ssl/client_local_client.key" },
                { "ssl.key.password", "abcdefgh" }

using simpleProducer (just modifying config and reporting error). Behaviour seems similar on 0.9.5 and 0.11.0.0-RC2 (tested on confluent.kafka 0.11.x branch, but it shouldn't change anything)

7|2017-06-29 23:53:40.027|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-29 23:53:40.083|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-29 23:53:40.084|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-29 23:53:40.084|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-29 23:53:40.143|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Broker SSL certificate verified
7|2017-06-29 23:53:40.238|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
3|2017-06-29 23:53:40.245|rdkafka#producer-1|FAIL| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
3|2017-06-29 23:53:40.246|rdkafka#producer-1|FAIL| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:
ssl://DESKTOP-LNQ6K3V:9093/0: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
ssl://DESKTOP-LNQ6K3V:9094/1: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:
7|2017-06-29 23:53:40.593|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: Broker SSL certificate verified
7|2017-06-29 23:53:40.597|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified

I can produce messages normally without other error

With only one broker up when lauching the app, the handshake failed does not appear

7|2017-06-30 00:06:38.758|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-30 00:06:38.764|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-30 00:06:38.764|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-30 00:06:38.764|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-30 00:06:38.802|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:06:38.867|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
3|2017-06-30 00:06:39.780|rdkafka#producer-1|FAIL| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Connect to ipv4#127.0.0.1:9093 failed: Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée.

ssl://localhost:9093/bootstrap: Connect to ipv4#127.0.0.1:9093 failed: Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée.

Making the other broker up, an any other connection/disconnection won't produce error - only at startup, and not always same error. Below 4 consecutive run with the two brokers alive, the error SSL handshake failed sometimes comes with a Receive failed, and sometimes no error at all:

$ dotnet run
7|2017-06-30 00:13:50.015|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-30 00:13:50.020|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-30 00:13:50.021|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-30 00:13:50.021|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-30 00:13:50.046|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Broker SSL certificate verified
ssl://localhost:9094/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
7|2017-06-30 00:13:50.062|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: Broker SSL certificate verified
7|2017-06-30 00:13:50.063|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
7|2017-06-30 00:13:51.067|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
q
7|2017-06-30 00:14:34.851|rdkafka#producer-1|DESTROY| [thrd:app]: Terminating instance
7|2017-06-30 00:14:34.851|rdkafka#producer-1|DESTROY| [thrd:main]: Destroy internal
7|2017-06-30 00:14:34.851|rdkafka#producer-1|DESTROY| [thrd:main]: Removing all topics

$ dotnet run
7|2017-06-30 00:14:38.129|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-30 00:14:38.134|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-30 00:14:38.134|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-30 00:14:38.134|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-30 00:14:38.149|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
ssl://localhost:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
ssl://localhost:9094/bootstrap: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:
2/2 brokers are down
7|2017-06-30 00:14:39.172|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:14:39.180|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:14:39.192|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: Broker SSL certificate verified
7|2017-06-30 00:14:39.201|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
q
7|2017-06-30 00:15:51.985|rdkafka#producer-1|DESTROY| [thrd:app]: Terminating instance
7|2017-06-30 00:15:51.985|rdkafka#producer-1|DESTROY| [thrd:main]: Destroy internal
7|2017-06-30 00:15:51.985|rdkafka#producer-1|DESTROY| [thrd:main]: Removing all topics

$ dotnet run
7|2017-06-30 00:15:56.326|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-30 00:15:56.331|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-30 00:15:56.331|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-30 00:15:56.331|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-30 00:15:56.355|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
ssl://localhost:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
ssl://localhost:9094/bootstrap: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:
2/2 brokers are down
7|2017-06-30 00:15:57.382|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:15:57.387|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:15:57.402|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
ssl://DESKTOP-LNQ6K3V:9093/0: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)
ssl://DESKTOP-LNQ6K3V:9094/1: Receive failed: .\ssl\ssl_lib.c:1075: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:
7|2017-06-30 00:15:57.673|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified
7|2017-06-30 00:15:57.678|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: Broker SSL certificate verified

no error :

$ dotnet run
7|2017-06-30 00:18:12.710|rdkafka#producer-1|SSL| [thrd:app]: Loading CA certificate(s) from file D:/kafka/ssl/ca-cert
7|2017-06-30 00:18:12.716|rdkafka#producer-1|SSL| [thrd:app]: Loading certificate from file D:/kafka/ssl/client_local_client.pem
7|2017-06-30 00:18:12.716|rdkafka#producer-1|SSL| [thrd:app]: Loading private key file from D:/kafka/ssl/client_local_client.key
7|2017-06-30 00:18:12.716|rdkafka#producer-1|SSLPASSWD| [thrd:app]: Private key file "D:/kafka/ssl/client_local_client.key" requires password
rdkafka#producer-1 producing on test2. q to exit.
7|2017-06-30 00:18:12.734|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9094/bootstrap]: ssl://localhost:9094/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:18:12.742|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: Broker SSL certificate verified
7|2017-06-30 00:18:12.751|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9093/0]: ssl://DESKTOP-LNQ6K3V:9093/0: Broker SSL certificate verified
7|2017-06-30 00:18:12.757|rdkafka#producer-1|SSLVERIFY| [thrd:ssl://DESKTOP-LNQ6K3V:9094/1]: ssl://DESKTOP-LNQ6K3V:9094/1: Broker SSL certificate verified

Will try to do more tests this we, but I assume there is some kind of race when trying to contact multiple brokers at startup.

Also, I don't have any more error with ssl.client.auth=none, and those are just debug messages (OnError does get called)

treziac commented 7 years ago

@edenhill did you try linking openssl 1.1.0 instead of 1.0.2?

edenhill commented 7 years ago

Thanks alot, this is very helpful and leads me to believe there is a concurrency problem with multiple simultaneos ssl sessions. Will investigate

Den 30 jun 2017 01:02 skrev "treziac" notifications@github.com:

@edenhill https://github.com/edenhill did you try linking openssl 1.1.0 instead of 1.0.2?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/61#issuecomment-312131142, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgCvlIwF_ewtcPdFQo0Stwf68WdqTjgks5sJC0GgaJpZM4MEHNK .

TheMidgardWatcher commented 7 years ago

Hi, @edenhill !

Any updates on this issue?

razinbouzar commented 7 years ago

Hi @edenhill, I'm working on connecting a .NET Client on a Windows host using SASL_SSL. Successfully able to connect with the Java client on a Unix host on SASL_SSL. Is there a configuration template for Windows hosts using this protocol?

In addition, tests are being run using the kafka-console-consumer bat file.

TheMidgardWatcher commented 7 years ago

Hi, @edenhill !

Any updates on this issue?

TheMidgardWatcher commented 7 years ago

Hi guys, any news about fixing this issue?

edenhill commented 7 years ago

This issue is a mix of SSL problems and feature request for SASL Kerberos support, the latter is explained here: https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka-on-Windows

The former should have its own issue.

TheMidgardWatcher commented 7 years ago

Hi there, @edenhill, is there any news about this "Ssl handshake failed" issue?

How can we force to fix this ASAP? This issue lasts from January ((

edenhill commented 7 years ago

@TheMidgardWatcher Can you try out librdkafka master and verify this fixes the problem? Artifacts are available here: https://ci.appveyor.com/project/edenhill/librdkafka/build/job/tdlfq2w6jii8t1y1/artifacts

Thanks

TheMidgardWatcher commented 7 years ago

@edenhill Seems like it works. But to be 100% sure - could you publish this package into pre-release nuget feed?

TheMidgardWatcher commented 7 years ago

Hi, @edenhill ! I'm testing your fix, and i don't see any SSL or Handshake Exceptions - that's great! We are using confluent 3.2.1 with 3 brokers.

PS: producer is horribly slow - 1k of avro records are sent in 15-20 minutes.

edenhill commented 7 years ago

SSL: That's great news, thanks!

Perf: try setting linger.ms to 100ms or more.

TheMidgardWatcher commented 7 years ago

I've started producer with linger.ms=1000

But, as we see from consumer log screenshot - producer sends ~3-5 messages in ~3-5 seconds almost 1 message/sec consumer_log

edenhill commented 7 years ago

I suggest focusing only on the producer if you are troubleshooting producer performance. Register a delivery report handler and measure the message rate there. To get an insight into what is happening under the hood, enable debug property with value msg,protocol and keep an eye on the number of messages per MessageSet (batch) and the size of ProduceRequests.

TheMidgardWatcher commented 7 years ago

Thanks for advice! I'll look into it. Handshake issue was fixed, and producer performance - is a story for another day.

UPD: producing was so slow because we produced each message synchronously :-| My bad. Now it sends >25k of messages in 5-6seconds

aleonidex commented 7 years ago

Hi @edenhill, could you pls speed up release of this handshake fix? We're really demanding it, kinda blocker for our team ((

edenhill commented 7 years ago

The final release will be a week or two, but we can get an RC up on NuGet mid this week.

aleonidex commented 7 years ago

Oh, that would be perfect!

we can get an RC up on NuGet mid this week

aleonidex commented 7 years ago

Hi @edenhill, any news??

mhowlett commented 7 years ago

sorry for the delay. I can assure you he's actively working on this... we're doing a lot of work to streamline librdkafka releases in general, and this is part of that effort.

TheMidgardWatcher commented 7 years ago

Hey there!

@edenhill or @mhowlett, will the next release be compatible with .Net Core 2.0?

mhowlett commented 7 years ago

I will test that, yes. related: #291.

buntyray commented 5 years ago

Kindly please share how this issue was resolved. I am using confluent 3.3.0 and I am seeing similar issue with my Python Avro producer and consumer

%3|1540503888.323|FAIL|rdkafka#producer-1| [thrd:ssl://xxxx.hostname.com:9093/bootstrap]: ssl://xxxx.hostname.com:9093/bootstrap: Connect to ipv4#x.x.x.x:9093 failed: Connection refused %3|1540503888.323|ERROR|rdkafka#producer-1| [thrd:ssl://xxxx.hostname.com:9093/bootstrap]: ssl://xxxx.hostname.com:9093/bootstrap: Connect to ipv4#x.x.x.x:9093 failed: Connection refused %3|1540503888.409|FAIL|rdkafka#producer-1| [thrd:ssl://yyyy.hostname.com:9093/bootstrap]: ssl://yyyy.hostname.com:9093/bootstrap: SSL handshake failed: s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log) .......................................

iMajna commented 4 years ago

@buntyray you maybe found a culprit?

confluentinc / confluent-kafka-dotnet

Support of SSL/Kerberos #61