Facing this issue while using authentication with TLS

[BajrangDubeyPersistent]

I followed all the steps mentioned in https://pulsar.apache.org/docs/en/security-tls-transport/ and https://pulsar.apache.org/docs/en/security-tls-authentication/ to use authentication using TLS but somehow whenever I make config changes and restart, broker containers get excited. while trying to look into the logs, I see following errors:

1. org.apache.pulsar.common.util.FileModifiedTimeUpdater - Unable to fetch lastModified time for file ca.cert.pem: java.nio.file.NoSuchFileException: /ca.cert.pem (address is correct but facing this error for all 3 certificates)

2. org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Execption while trying to refresh ssl Context Certificate loading error java.security.KeyManagementException: Certificate loading error

3. Will not attempt to authenticate using SASL (unknown error)

4. ERROR org.apache.pulsar.broker.PulsarService - java.lang.NullPointerException

I was expecting this to run fine but couldn,t find any solution on internet. If you could help me with what could be the issue, it'll be really helpful. I have generated certificate from openSSL as mentioned in the documentation for apache pulsar.

[nodece]

Maybe you use a relative path? This should use an absolute path.

[BajrangDubeyPersistent]

No, I have checked and made sure that the path for certificate files is absolute only

[nodece]

Could you provide how to reproduce this issue and the version information?

[BajrangDubeyPersistent]

I am using apachepulsar/pulsar-all:2.5.0 docker image to run pulsar. Everything is running well because I have tried all pulsar functions using this image only. While doing some changes in conf files taken from sijie's repo: https://github.com/streamnative/tgip/tree/master/episodes/001/conf and making changes to implement authentication using TLS as mentioned in official documentation https://pulsar.apache.org/docs/en/security-tls-transport/ and https://pulsar.apache.org/docs/en/security-tls-authentication/, I faced mentioned issues. The broker got exited as these changes were made and logs show these details for exited broker container:

1. ERROR org.apache.pulsar.common.util.FileModifiedTimeUpdater - Unable to fetch lastModified time for file /home/bajrang_dubey/my-ca/certs/ca.cert.pem: java.nio.file.NoSuchFileException: /home/bajrang_dubey/my-ca/certs/ca.cert.pem Same for all 3 certificates: broker.cert.pem, broker.key-pk8.pem, ca.cert.pem

2. After regular tries, it shows this: ERROR org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Execption while trying to refresh ssl Context Certificate loading error java.security.KeyManagementException: Certificate loading error at org.apache.pulsar.common.util.SecurityUtility.loadCertificatesFromPemFile(SecurityUtility.java:179) ~[org.apache.pulsar-pulsar-common-2.5.0.jar:2.5.0]

3. ERROR org.apache.pulsar.broker.PulsarService - java.lang.NullPointerException org.apache.pulsar.broker.PulsarServerException: java.lang.NullPointerException ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service. org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: java.lang.NullPointerException

4. Will not attempt to authenticate using SASL (unknown error)

These are 4 errors that I can see in broker container logs, couldn't find apt solution to it. Please help me if I am missing something here as I am stuck here for quite some time.

[nodece]

The apachepulsar/pulsar-all:2.5.0 version is too old, the Pulsar 2.10.0 has been released, you can try this image:

docker pull apachepulsar/pulsar-all:2.10.0

You can find any version here.

[BajrangDubeyPersistent]

Hi @nodece, apologies for the late response. I tried working on TLS with Pulsar 2.10.0 but the docker image seems corrupted as it showed some access error for data directory : https://stackoverflow.com/questions/72165829/apache-pulsar-zookeeper-unable-to-access-datadir-exiting-abnormally

So, to switch to more stable version, I am currently using 2.8.2 but facing the same errors in log. Is there some place else where I can find TLS authentication steps for apache pulsar as I am wondering if there is some issue with certificates generated and maybe I can try generating certificates for some place elsewhere.

Can you please help me with the errors posted as I can surely say that these are not the version related issue.

[nodece]

You can try to use cfssl to generate the TLS certificate.

echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - |cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"876000h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json
echo '{"CN":"broker","hosts":["localhost"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem - | cfssljson -bare broker
echo '{"CN":"client","hosts":["localhost"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem - | cfssljson -bare client

# Java requires pk8 format key
openssl pkcs8 -topk8 -inform PEM -outform PEM -in client-key.pem -out client-key-pk8.pem -nocrypt
openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker-key.pem -out broker-key-pk8.pem -nocrypt

[BajrangDubeyPersistent]

Hi Nodece,

Sorry but changing certification didn't work. I am not sure why is it fetching lastModifiedTime and is unable to find file at the exact location it is supposed to find. Shows FileNotFound Exception, it is in absolute file too that I checked. Path starts as: tlsCertificateFilePath= /home/bajrang_dubey/my-ca/broker.cert.pem

[nodece]

If you are using the Pulsar with docker, maybe you need to check the user permission.

[BajrangDubeyPersistent]

I have root privileges too and have also tried using TLS in root user but it didn't work. I also used chmod to change permission to 777 to make sure it is readable. Still for some reason, it is trying to fetch lastModified time and then is unable to locate file too.

[nodece]

Could you provide a docker image to reproduce this issue?

[BajrangDubeyPersistent]

apachepulsar/pulsar-all 2.8.2 using this image with documentation from https://pulsar.apache.org/docs/security-tls-transport/

[nodece]

Sorry, I cannot reproduce this issue by using apachepulsar/pulsar-all:2.8.2.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.