search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.27k stars 3.59k forks source link

TLS mutual Authentication with keystore configuration on Linux OS fails to establish a successful connection #20249

Open ragaur-tibco opened 1 year ago

ragaur-tibco commented 1 year ago

Search before asking

Version

Java Client(2.11.0)-->broker(2.11.0) Broker instance OS: Amazon Linux 2023 Java App running OS : RHEL and Ubuntu

Minimal reproduce step

Configure the Pulsar instance for mTLS authentication using Keystore. Then create the Pulsar client using the following sample code.

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar+ssl://broker.example.com:6651/")
    .useKeyStoreTls(true)
    .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    .tlsTrustStorePassword("clientpw")
    .allowTlsInsecureConnection(false)
    .enableTlsHostnameVerification(false)
    .authentication(
            "org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
            "keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
    .build();

What did you expect to see?

It should get succeeded with the correct keystore path configuration for Linux.

What did you see instead?

trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
2023-05-03T11:28:07,645 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2023-05-03T11:28:07,646 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024

2023-05-03T11:28:08,252 DEBUG [pulsar-client-io-1-1] org.apache.pulsar.common.util.SecurityUtility - Already instantiated Bouncy Castle provider BCFIPS
2023-05-03T11:28:08,299 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
    at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
    at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
    at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
    at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
    at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
    at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
    at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
2023-05-03T11:28:08,304 WARN  [pulsar-client-io-1-1] org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 : java.lang.NullPointerException
2023-05-03T11:28:08,407 WARN  [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://public/default/test-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms
2023-05-03T11:28:08,407 DEBUG [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.ConnectionPool - Connection for ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 not found in cache
2023-05-03T11:28:08,410 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
    at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
    at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
    at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
    at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
    at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
    at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
    at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

Anything else?

As you can observe the debug logs, the keystore path we are providing is not taking into effect and it is being set to null and throwing the file not found error.

Note: Same source code is working fine with windows machine.

Are you willing to submit a PR?

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.

tisonkun commented 1 year ago

This can be related to https://github.com/apache/pulsar/pull/19483. You can try out 2.11.1 or 3.0.0.

cc @nodece

nodece commented 1 year ago

I cannot reproduce this issue, could you check your keyStorePath?

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.