search

commercetest / pepper-box

Pepper-Box is kafka load generator plugin for jmeter. It produces kafka messages of type plain text (JSON, XML, CSV or any other custom format) as well as java serialized objects.
http://pepperbox.gslab.com/
Apache License 2.0
3 stars 3 forks source link

Add support for SASL #1

Closed julianharty closed 6 years ago

julianharty commented 6 years ago

Kafka optionally supports various security mechanisms. One is SASL http://kafka.apache.org/documentation.html#security_sasl This requires additional parameters to be passed by pepper-box to Kafka. It also affects the contents of broker information stored in and returned by Zookeeper (which I'll cover in a separate issue).

I'd like to add sufficient support for SSL_SASL to at least satisfy the testing needs I have. I don't know all the nuances of other permutations to have confidence the support would work for every permutation or variation of SSL_SASL

julianharty commented 6 years ago

See https://developer.ibm.com/hadoop/2017/04/10/kafka-security-mechanism-saslplain/ for more useful information.

julianharty commented 6 years ago

And the definitive reference http://kafka.apache.org/documentation.html#security_sasl_plain_clientconfig which includes:

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="alice" \
    password="alice-secret";

    security.protocol=SASL_SSL
    sasl.mechanism=PLAIN

We'll also need to add support in pepper-box for SSL, as the documentation states: "_SASL may be used with PLAINTEXT or SSL as the transport layer using the security protocol SASL_PLAINTEXT or SASL_SSL respectively. If SASL_SSL is used, then SSL must also be configured._"

Note: the servers also need configuring, which is described in the Apache Kafka docs.

julianharty commented 6 years ago

http://kafka.apache.org/documentation.html#security_configclients "If client authentication is not required in the broker, then the following is a minimal configuration example:" i.e. this is the minimum we have to include, additional parameters may be necessary.

security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=test1234

Note we'll need and use security.protocol=SASL_SSL

julianharty commented 6 years ago

Here are some notes based on the challenges I had in discovering mis-configurations of what turned out to be working software. Perhaps they'll help me and others in future who face the challenge of getting this software communicating with Kafka brokers that require SASL_SSL.

Context: I spent several hours trying to understand why pepper-box wasn't able to send messages successfully. Eventually I resolved the problem when I corrected the password being used as part of the sasl.jaas.config which was wrong for the server I wanted to connect to. Even the debug logs didn't contain many clues.

Before diving into details:

Here's an extract from the log (which I've edited to remove potentially sensitive information). There are 2 kafka nodes. The pattern is similar and repeats many times in the log of an attempted handshake followed by a Java end of file exception: null

2018-01-16 21:04:57,340 DEBUG o.a.k.c.NetworkClient: Initialize connection to node -2 for sending metadata request
2018-01-16 21:04:57,340 DEBUG o.a.k.c.NetworkClient: Initiating connection to node -2 at kafka-node2:9093.
2018-01-16 21:04:57,343 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Set SASL client state to SEND_HANDSHAKE_REQUEST
2018-01-16 21:04:57,344 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Creating SaslClient: client=null;service=kafka;serviceHostname=kafka-node2;mechs=[PLAIN]
2018-01-16 21:04:57,495 DEBUG o.a.k.c.m.Metrics: Added sensor with name node--2.bytes-sent
2018-01-16 21:04:57,496 DEBUG o.a.k.c.m.Metrics: Added sensor with name node--2.bytes-received
2018-01-16 21:04:57,496 DEBUG o.a.k.c.m.Metrics: Added sensor with name node--2.latency
2018-01-16 21:04:57,497 DEBUG o.a.k.c.n.Selector: Created socket with SO_RCVBUF = 33696, SO_SNDBUF = 132288, SO_TIMEOUT = 0 to node -2
2018-01-16 21:04:57,497 DEBUG o.a.k.c.NetworkClient: Completed connection to node -2.  Fetching API versions.
2018-01-16 21:04:57,592 DEBUG o.a.k.c.n.SslTransportLayer: SSL handshake completed successfully with peerHost 'kafka-node1' peerPort 9093 peerPrincipal 'CN=x, OU=x, O="x", L=x, ST=x, C=x' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256'
2018-01-16 21:04:57,606 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE
2018-01-16 21:04:57,758 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Set SASL client state to INITIAL
2018-01-16 21:04:57,759 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Set SASL client state to INTERMEDIATE
2018-01-16 21:04:57,820 DEBUG o.a.k.c.n.SslTransportLayer: SSL handshake completed successfully with peerHost 'kafka-node2' peerPort 9093 peerPrincipal 'CN=x, OU=x, O="x", L=x, ST=x, C=x' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256'
2018-01-16 21:04:57,821 DEBUG o.a.k.c.s.a.SaslClientAuthenticator: Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE
2018-01-16 21:04:57,911 DEBUG o.a.k.c.n.Selector: Connection with kafka-node1/x.x.x.x disconnected
java.io.EOFException: null
        at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:493) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:81) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.receiveResponseOrToken(SaslClientAuthenticator.java:242) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:185) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:71) ~[pepper-box-1.0.jar:?]
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:350) [pepper-box-1.0.jar:?]
        at org.apache.kafka.common.network.Selector.poll(Selector.java:303) [pepper-box-1.0.jar:?]
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:349) [pepper-box-1.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:225) [pepper-box-1.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:126) [pepper-box-1.0.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
2018-01-16 21:04:57,912 DEBUG o.a.k.c.NetworkClient: Node -1 disconnected.

Eventually I stop the test (which is hanging in the GUI) and see the following pattern in the jmeter log:

2018-01-16 21:05:08,088 INFO o.a.j.g.a.Start: Stopping test
2018-01-16 21:05:08,108 INFO o.a.j.t.JMeterThread: Stopping: DoLittle 3.2 1-1
2018-01-16 21:05:08,108 WARN o.a.j.t.JMeterThread: Interrupting: DoLittle 3.2 1-1 sampler: Java Request
2018-01-16 21:05:08,108 WARN o.a.j.t.JMeterThread: No operation pending
2018-01-16 21:05:08,109 ERROR c.g.p.s.PepperBoxKafkaSampler: Failed to send message

There were no obvious error messages, just the lack of progress and the end of file exception.

julianharty commented 6 years ago

Here are some online posts I found for seemingly similar errors: https://wwija.com/computer-internet-technology/3601843_kafka-0-10-sasl-plain-producer-timeout.html (they were trying to connect using python with a slightly older version of the Kafka libraries) https://stackoverflow.com/questions/41961132/configure-kafka-client-to-connect-with-issued-ssl-key-cert a similar EOF exception, for different, yet perhaps similar, reasons. https://q-a-assistant.info/computer-internet-technology/configure-kafka-client-to-connect-with-issued-ssl-key-cert/3965543 which seems very similar to the problem I was experiencing, identified as a mismatch and incompatibility between releases of Kafka: "It turns out that my Kafka cluster (Heroku add-on) wasn't actually running 0.10.1.1, it was running 0.10.0.1. The two seem to have incompatible consumer APIs." https://stackoverflow.com/questions/39521691/kafka-authentication-producer-unable-to-connect-producer which summarizes a longer conversation recorded in an email archive where the person had an incorrect password in their configuration. http://community.cloudera.com/t5/Data-Ingestion-Integration/Timeout-Error-When-Using-kafka-console-consumer-and-kafka/m-p/58061 contains a saga of similar proportions together with useful commands to help debug the connectivity and configuration https://groups.google.com/forum/#!topic/confluent-platform/1AnoKUIw91Y seems to be causes by a similar mis-configuration http://mail-archives.apache.org/mod_mbox/kafka-users/201609.mbox/%3CCAHX2Snk11vg7DXNVUr9oE97ikFSQUoT3kBLAxYymEDj7E14XrQ@mail.gmail.com%3E unlike my experience their logs include a message stating there's a problem with the username or password.

julianharty commented 6 years ago

At least for my use, I think I've got the essentials working for SASL_SSL so I'll close this issue. We can open a new one for any additional functionality, fixes, or updates.