Can't connect to broker with SSL_SASL

[ledocool]

• PHP version: 7.3
• librdkafka version: 0.9.3
• php-rdkfaka version: 3.1.0-dev
• OS used: Debian 9

I can't seem to connect to remote kafka broker when using ssl_sasl. Whenever I try I get this: "../ssl/ssl_rsa.c:615: error:140DC002:SSL routines:use_certificate_chain_file:system lib: "

My config:

    'kafka' => [
        'common' => [
            'metadata.broker.list' => 'my.nice.broker.com:9095',
//            'bootstrap.servers' => ['my.nice.broker.com:9095'],
            'group.id' => 'GroupId',
            'security.protocol' => 'SASL_SSL',
            'sasl.mechanisms' => "SCRAM-SHA-256",
            'ssl.key.location' => __DIR__ . '/Certificates/client.key',
            'ssl.certificate.location' =>  __DIR__ . '/Certificates/client.cer.pem',
            'ssl.ca.location' =>__DIR__ . '/Certificates/',
            'ssl.key.password' => 'herebedragons',
            'sasl.password' => 'test1',
            'sasl.username' => 'test1',
        ],    
    ],

Certificate is self-signed, was extracted from JKS file.

Thinking it is a wrong config I tried removing ssl.key.location and ssl.certificate.location. It somewhat worked in a sense that I don't get an exception, but I'm still unable to draw messages in (consume returns null while there should be messages on the other side). rd_kafka_errno() returns me error 115 which I couldn't get much info on. And consumeStop function seem to be hanging php for good.

[Steveb-p]

You might try using a newer version of librdkafka. I'd suggest 0.11.6 or 1.0.0, since those have been kept relatively up to date with patches.

[ledocool]

Didn't help, sadly. Oddly enough, I noticed that high level consumer returns RD_KAFKA_RESP_ERR__TIMED_OUT (kafka 1.0.0, built from source). Should I be digging into my connection?

[nick-zh]

@ledocool RD_KAFKA_RESP_ERR__TIMED_OUT is sometimes an indicator for connection issues, yes. Were you able to resolve your issue in the meantime?

[ledocool]

@nick-zh yes, fiddling with certificates helped. I'm posting my configuration later because I think this may be of help to someone.

[nick-zh]

@ledocool very cool, thank you!

[nick-zh]

@ledocool i am closing this, but feel free to still add your configuration, could be helpful to others :+1:

[ledocool]

@nick-zh oh, I forgot about it!

'kafka' => [
        'common' => [
            'metadata.broker.list' => 'my.nice.server:9095',
            'group.id' => 'groupid',
            'security.protocol' => 'SASL_SSL',
            'sasl.mechanisms' => "SCRAM-SHA-256",
            'ssl.certificate.location' =>  __DIR__ . '/Certificates/all/client.cer.pem',
            'ssl.ca.location' => __DIR__ . '/Certificates/all/client.cer.pem',
            'ssl.key.password' => 'keystoragepassword',
            'sasl.password' => 'saslpassword',
            'sasl.username' => 'sasllogin',
        ],   
]

Key part is:

'ssl.certificate.location' =>  __DIR__ . '/Certificates/all/client.cer.pem',
'ssl.ca.location' => __DIR__ . '/Certificates/all/client.cer.pem',

We're using self-signed certificate (extracted from jks storage) for our test machine.

[nick-zh]

@ledocool thank you :+1:

[jvarugh]

@ledocool

I'm doing the exact same this as you right now, but I'm trying to produce to brokers that I do not manage. I was given a truststore.jks file and its password.

I've extracted that into a password protected .pem using keytool -> pkcs12, then openssl -> pem, but still cannot seem to resolve the handshake failure.

1. Why are your ca-cert and regular cert the same?
2. Is your PEM still password protected? If yes, is that what the "ssl.key.password" is and why don't you need a key location?

Thanks,

[ledocool]

Why are your ca-cert and regular cert the same?

As far as I remember we used self-signed certificate and this is why I had to use same certificate in both locations. I remember I reached that via trial and error, so can't really tell why exactly it should be like this. My take would be that this pem is sufficient for validating itself.

Is your PEM still password protected? If yes, is that what the "ssl.key.password" is and why don't you need a key location?

I believe so. And yes, "ssl.key.password" contains password to that jks store. The pass could be changed during extraction, if I remember correctly, but we used same pass for both jks and extracted certs.

Also note that I used application called KeyStore Explorer (5.4.1) to extract root cert. It seems to me that this app produces slightly different result from what console utils would do and somehow (magically, dare I say) cert extracted by KeyStore is much more welcomed by kafka. At least this is what I came to after people asked me to help with this.

[jvarugh]

@ledocool

KeyStore actually helped me. I found the issue by analyzing the truststore.jks. It did not contain certs for several of the brokers listed in the documentation, thus the failure.

Thanks for your help and the quick reply. Cheers.

[carlosbuenosvinos]

Hi all,

I have two .p12 certificates for connecting using Java clients (keystore.p12 that goes into the "ssl.keystore.location" configurations and the truststore.p12 that goes into the "ssl.truststore.location"). We want to try with Python or PHP and "truststore" is not supported by the librdkafka library.

Is there a way to extract the certificates needed from the two java focus .p12 ones? @ledocool @jvarugh Thanks in advance!

[ledocool]

Hi all,

I have two .p12 certificates for connecting using Java clients (keystore.p12 that goes into the "ssl.keystore.location" configurations and the truststore.p12 that goes into the "ssl.truststore.location"). We want to try with Python or PHP and "truststore" is not supported by the librdkafka library.

Is there a way to extract the certificates needed from the two java focus .p12 ones? @ledocool @jvarugh Thanks in advance!

There was an application named key store explorer i mentioned earlier. I also mentioned version i used. I used that to extract from java truststore. I also believe you could do it calling openssl directly, but its a bit harder.

[carlosbuenosvinos]

thanks @ledocool for the quick answer! I have the KeyStore explorer and I can explore both .p12 certificates. Do you know exactly what pem certificates do I need to extract from the p12?

Based on your old previous comment:

• ssl.ca.location certificate from keystore.p12?
• ssl.key.location certificate from keystore.p12?
• ssl.keystore.location certificate from truststore.p12?

Is that right?

[ledocool]

I'm afrad I'm not that well-versed in the cerificate world. You might have to experiment. We had one store on our hands and we extracted all three from it. It was jks store, maybe - cant recall clearly.

[ledocool]

thanks @ledocool for the quick answer! I have the KeyStore explorer and I can explore both .p12 certificates. Do you know exactly what pem certificates do I need to extract from the p12?

Based on your old previous comment:

• ssl.ca.location certificate from keystore.p12?
• ssl.key.location certificate from keystore.p12?
• ssl.keystore.location certificate from truststore.p12?

Is that right?

More likely ca is in the truststore, while pem and key are in the keystore, tho. But its just a hunch.