search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
68 stars 861 forks source link

Support of SSL/Kerberos #61

Closed thexixx closed 7 years ago

thexixx commented 7 years ago

Hello!

Could someone tell me if this lib supports SSL/Kerberos? At the momoent I can't find any .Net Kafka Client which is supporting SSL and(or) Kerberos.

edenhill commented 7 years ago

Hi!

Yes, the upcoming first version of Confluent's .NET client will support both SSL and SASL.

For instructions on how to use SSL, see See https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka

For SASL authentication the support is platform dependent:

The client will be released very soon.

simplesteph commented 7 years ago

@edenhill will the client be released with Confluent 3.2.0 or independently? Is there a version we can test in the meantime?

edenhill commented 7 years ago

@simplesteph It will be released with Confluent 3.2.0 You could try the latest preview here: https://www.nuget.org/packages/Confluent.Kafka/0.9.4-preview4

TheMidgardWatcher commented 7 years ago

In continuation of my Issue ah-/rdkafka-dotnet#112 I should say that i'm getting same ssl errors: 3|2017-03-01 15:23:23.161|Test@localhost#consumer-2|FAIL| [thrd:ssl://x.x.x.x:9093/1]: ssl://x.x.x.x:9093/1: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)

Error: Local_Ssl ssl://x.x.x.x:9093/1: SSL handshake failed: .\ssl\s3_both.c:408: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log)

Should i open new issue here?

edenhill commented 7 years ago

@TheMidgardWatcher Did you check the broker logs as instructed?

TheMidgardWatcher commented 7 years ago

@edenhill I'm not sure what you mean, but yes - we checked logs from brokers and also we checked java ssl debug logs. Actually I think I have already written about the results in previous issue.

simon406 commented 7 years ago

@edenhill I would like to use SASL/PLAIN for authentication. Do we have an example on how to do that by using Confluent.Kafka .net client? Thanks

edenhill commented 7 years ago

Depends on your platform. SASL/PLAIN is currently not supported on Windows, but works on OSX and Linux, etc.

What platform are you on?

simon406 commented 7 years ago

@edenhill I will run Kafka on Linux, but publisher and consumer which use Confluent.Kafka .net client will be running on windows servers. Would SASL/PLAIN work in this case?

Thanks

edenhill commented 7 years ago

@simon406 Sorry to say that SASL/PLAIN is not yet implemented for Windows (exists on Linux and OSX).

You could use SSL authentication or SASL Kerberos though (Windows AD)

simon406 commented 7 years ago

Thanks @edenhill

I am trying to integrate Kafka authentication with a token-based authentication web api. My plan is to have my own implementation of javax.security.sasl.SaslServer which would send tokens to the auth web api for authentication. This might not be possible without SASL/PLAIN.

dimastatz commented 7 years ago

Could you please add a code sample which shows KafkaConsumer with enabled SSL

TheMidgardWatcher commented 7 years ago

Could you please add a code sample which shows KafkaConsumer with enabled SSL

This would be nice, because i'm still getting this annoying SSL Handshake exception

TheMidgardWatcher commented 7 years ago

Any updates on Using SSL on windows? I suggest someone to change type of this issue to a bug at least.

TheMidgardWatcher commented 7 years ago

Hi there! Just checked topic consumption by ssl configured kafka-console-consumer on local and remote kafka environments. As a result - both environments was successfully read by it. But locally built librdkafka_example didn't read from any environment. So reason of 'SSL handshake failed' definitely not in bad environments or its configuration, and not in confluent-kafka-dotnet wrapper. But i should say that producer works fine - without any problems on both local and remote kafka environments.

TheMidgardWatcher commented 7 years ago

Also verified self-signed certificates on local machine - same issue Consume failed: Local: Communication failure with broker

edenhill commented 7 years ago

If you enable SSL debugging in the broker (-Djavax.net.debug=all) and look at stderr, does it tell you anything interesting about the connecting client?

TheMidgardWatcher commented 7 years ago

Should I look something special? All output looks fine. I got this lines at the end of output:

kafka-network-thread-0-ListenerName(SSL)-SSL-4, READ: TLSv1.2 Alert, length = 26
Padded plaintext after DECRYPTION:  len = 2
0000: 01 00                                              ..
kafka-network-thread-0-ListenerName(SSL)-SSL-4, RECV TLSv1.2 ALERT:  warning, close_notify
kafka-network-thread-0-ListenerName(SSL)-SSL-4, closeInboundInternal()
kafka-network-thread-0-ListenerName(SSL)-SSL-4, closeOutboundInternal()
kafka-network-thread-0-ListenerName(SSL)-SSL-4, SEND TLSv1.2 ALERT:  warning, description = close_notify
Padded plaintext before ENCRYPTION:  len = 2
0000: 01 00                                              ..
kafka-network-thread-0-ListenerName(SSL)-SSL-4, WRITE: TLSv1.2 Alert, length = 26
kafka-network-thread-0-ListenerName(SSL)-SSL-4, called closeOutbound()
kafka-network-thread-0-ListenerName(SSL)-SSL-4, closeOutboundInternal()
[Raw write]: length = 31
edenhill commented 7 years ago

Okay, so that looks like it might be the client closing the connect.

Is debug=security,broker,protocol telling you anything on the client?

TheMidgardWatcher commented 7 years ago

This what i got:

LOG-7-TOPBRK: [thrd::0/internal]: :0/internal: Topic topic.test [0]: joining broker (rktp 0000024322141700)
Consume failed: Local: Communication failure with broker
LOG-7-UPDATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/bootstrap: NodeId changed from -1 to 0
LOG-7-UPDATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Name changed from ssl://localhost:9093/bootstrap to ssl://localhost:9093/0
LOG-7-LEADER: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Mapped 1 partition(s) to broker
LOG-7-STATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Broker changed state UP -> UPDATE
LOG-7-TOPBRK: [thrd::0/internal]: :0/internal: Topic topic.test [0]: leaving broker (0 messages in xmitq, next leader ssl://localhost:9093/0, rktp 0000024322141700)
LOG-7-SEND: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Sent MetadataRequest (v0, 44 bytes @ 0, CorrId 2)
LOG-7-STATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Broker changed state UPDATE -> UP
LOG-7-TOPBRK: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Topic topic.test [0]: joining broker (rktp 0000024322141700)
LOG-7-RECV: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Received MetadataResponse (v0, 78 bytes, CorrId 2, rtt 16.00ms)
LOG-7-DESTROY: [thrd:app]: Terminating instance
LOG-7-DESTROY: [thrd:main]: Destroy internal
LOG-7-DESTROY: [thrd:main]: Removing all topics
LOG-7-TERMINATE: [thrd::0/internal]: :0/internal: Handle is terminating: failed 0 request(s) in retry+outbuf
LOG-7-BROKERFAIL: [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error)
LOG-7-STATE: [thrd::0/internal]: :0/internal: Broker changed state UP -> DOWN
LOG-7-TOPBRK: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Topic topic.test [0]: leaving broker (0 messages in xmitq, next leader (none), rktp 0000024322141700)
LOG-7-TOPBRK: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Topic topic.test [0]: no next leader, failing 0 message(s) in partition queue
LOG-7-TERMINATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Handle is terminating: failed 0 request(s) in retry+outbuf
LOG-7-BROKERFAIL: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: failed: err: Local: Broker handle destroyed: (errno: No error)
LOG-7-STATE: [thrd:ssl://localhost:9093/bootstrap]: ssl://localhost:9093/0: Broker changed state UP -> DOWN
edenhill commented 7 years ago

There are no SSL errors in the above log, it succesfully connects to broker 0 and manages to acquire Metadata, which means the SSL layer is working.

After that (there are no timestamps so hard to say how long) it seems like you are bringing down the consumer (Terminating instance), possibly triggered by Consume failed: Local: Communication failure with broker.

Could you provide the full logs from client startup to shutdown?

TheMidgardWatcher commented 7 years ago

Do you mean client logs or from broker?

edenhill commented 7 years ago

Client

Den 16 maj 2017 5:37 em skrev "TheMidgardWatcher" <notifications@github.com

:

Do you mean client logs or from broker?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/61#issuecomment-301822335, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgCvvYt2RvA6keeLpR9dpWm452QKQVDks5r6cLAgaJpZM4MEHNK .

TheMidgardWatcher commented 7 years ago

Here you are:

client.log.txt

edenhill commented 7 years ago

There are no errors in the log, which makes the Consume error weird. I'm guessing you are shutting down your app when you get a consume error, can you let it continue to run to see if it works despite the consume error, possibly providing more logs?

edenhill commented 7 years ago

It would also be great if you could provide the source code of your program, the parts interacting with the Kafka library

TheMidgardWatcher commented 7 years ago

I'm using only librd sources, no other apps, just console command. I've compiled it locally with vs 2013. Windows 10 Enterprise x64 OpenSSL 1.0.2k Latest Kafka 2.12-0.10.2.1 Zookeeper 3.4.10

Console command I'm running: C:\Sources\librdkafka-master\win32\outdir\v120\x64\Release>rdkafka_example -C -t topic.test -b localhost:9093 -X security.protocol=ssl -X ssl.ca.location=d:\\Certificates\\TEST\\ca-cert -X ssl.certificate.location=d:\\Certificates\\TEST\\client.pem -X ssl.key.location=d:\\Certificates\\TEST\\client.key -X ssl.key.password=123456 -X debug=security,broker,protocol -p 0 -o 0 2>client.log Full console output: consumer.log.txt

TheMidgardWatcher commented 7 years ago

Hi there, any updates on this Issue?

edenhill commented 7 years ago

This looks like a problem with rdkafka_example..

Is topic.test an existing topic? Can you try -L (metadata list) or -P (produce) instead of -C ?

TheMidgardWatcher commented 7 years ago

Is topic.test an existing topic? Can you try -L (metadata list) or -P (produce) instead of -C ?

Yes it exists. -L works fine.

edenhill commented 7 years ago

Okay, good, that means your SSL configuration is working as it should. I think you can disregard the consumer error (please file a librdkafka issue for it!), it is an example tool bug.

TheMidgardWatcher commented 7 years ago

Ok, I'll start issue in librd repo. But what to do next with "SSL Handshake" error?

edenhill commented 7 years ago

rdkafka_example isnt showing SSH Handshake error, right? Do you get the SSH Handshake error with dotnet using the exact same configuration you use for rdkafka_example?

TheMidgardWatcher commented 7 years ago

Hi, @edenhill !

After my testing i found that:

  1. On my local kafka everything works fine - both .net consumer and kafka's console consumer.
  2. On remote confluent 3.2.0 platform - .net client throws 'SSL handshake' and 'Shutdown while init' errors, but kafka's console consumer works properly!

I'm going crazy with this issue...

edenhill commented 7 years ago

Can you provide logs from the non-working client with debug=security,protocol,broker?

The client might be unable to verify the broker's key due to no default ca cert locations, specifying the ca.location might do the trick.

This commit fixes that : https://github.com/edenhill/librdkafka/commit/10f9de510cf088381d7199bea8bd7a65b97ab5f1#diff-226483c4c83ab9400939e815bcb19564

Den 24 maj 2017 12:49 skrev "TheMidgardWatcher" notifications@github.com:

Hi, @edenhill https://github.com/edenhill !

After my testing i found that:

  1. On my local kafka everything works fine - both .net consumer and kafka's console consumer.
  2. On remote confluent 3.2.0 platform - .net client throws 'SSL handshake' and 'Shutdown while init' errors, but kafka's console consumer works properly!

I'm already went crazy with this issue...

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/61#issuecomment-303689196, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgCvow8JCiVH2fr9mo20svPmPD2rgEiks5r9As_gaJpZM4MEHNK .

TheMidgardWatcher commented 7 years ago

I'm passing ssl.ca.location. You can see it in attached config dump. client_config_dump.txt client_debug_log.txt

edenhill commented 7 years ago

Thanks,

what if you run the openssl client tool, it should give you some more information what is wrong: openssl s_client -connect <broker>:<port>

TheMidgardWatcher commented 7 years ago

openssl output:


CONNECTED(000001A8)
depth=1 C = XX, L = YYY, O = ZZZ
verify error:num=19:self signed certificate in certificate chain
79952:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:```
edenhill commented 7 years ago

ah! https://stackoverflow.com/questions/12180552/openssl-error-self-signed-certificate-in-certificate-chain

TheMidgardWatcher commented 7 years ago

That's strange, because if i call openssl s_client -CAfile D:\Certificates\Sandbox\ca.crt -cert D:\Certificates\Sandbox\client.crt -key D:\Certificates\Sandbox\client.key -connect broker1:9093 I got:

CONNECTED(000001AC)
depth=1 C = XX, L = YYY, O = ZZZ
verify return:1
depth=0 CN = *.zzz.com
verify return:1
[...]
Verify return code: 0 (ok)
edenhill commented 7 years ago

Looking at the logs it seems to be able to connect with SSL to broker1 and broker3, but not broker2. Try running the s_client thing for all three brokers and see if you get different results for broker2

TheMidgardWatcher commented 7 years ago

All brokers are returning: Verify return code: 0 (ok) That's only if i pass -cert and -key

edenhill commented 7 years ago

Try running the Kafka client with just one bootstrap broker at the time, and see if there is a difference. If it manages to connect and acquire metadata it means the config/certs are correct and it would indicate an issue with reusing the same SSL context.

TheMidgardWatcher commented 7 years ago

Nothing changed. Here is the logs and config: client_config_dump.txt client_debug_log.txt

edenhill commented 7 years ago

Is this with Kafka v 0.10.2.1 and librdkafka v0.9.5?

TheMidgardWatcher commented 7 years ago

As i know it is: Confluent 3.2.0 Kafka 0.10.2.0

And librdkafka v0.9.5

edenhill commented 7 years ago

If you enable SSL debugging on the broker, will it tell you anything interesting? Add JVM flags: -D java.security.debug=all

TheMidgardWatcher commented 7 years ago

Sorry for delay. Here is error from broker logs (if u need - I can send you full logs from 3 brokers):

[2017-05-25 18:20:15,693] DEBUG SSLEngine.closeInBound() raised an exception. (org.apache.kafka.common.network.SslTransportLayer:733)
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
    at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:731)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:314)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:69)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:350)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
    at kafka.network.Processor.poll(SocketServer.scala:494)
    at kafka.network.Processor.run(SocketServer.scala:432)
    at java.lang.Thread.run(Thread.java:745)
[2017-05-25 18:20:15,694] DEBUG Connection with workstation/10.6.XX.XX disconnected (org.apache.kafka.common.network.Selector:375)
javax.net.ssl.SSLHandshakeException: certificate verify message signature error
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:382)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:243)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:69)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:350)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
    at kafka.network.Processor.poll(SocketServer.scala:494)
    at kafka.network.Processor.run(SocketServer.scala:432)
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: certificate verify message signature error
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
    at sun.security.ssl.ServerHandshaker.clientCertificateVerify(ServerHandshaker.java:1658)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:293)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
    at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:336)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:417)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:270)
    ... 6 more
edenhill commented 7 years ago

Did you look into the exception being thrown?

[2017-05-25 18:20:15,694] DEBUG Connection with workstation/10.6.XX.XX disconnected (org.apache.kafka.common.network.Selector:375)
javax.net.ssl.SSLHandshakeException: certificate verify message signature error
TheMidgardWatcher commented 7 years ago

Our DevOps had some tests. And that what they reported:

  1. Java driver with enabled -D java.security.debug=all on brokers doesn't throw any ssl errors.
  2. openssl tool - works fine.
  3. librdkafka_example compiled under linux OS - works fine.

So they definitely sure that problem not in Brokers or certificates. Also python librdkafka - works fine too.