Client certificate signature issue when using netty-tcnative-boringssl-static with IBM J9 JDK

manikantag commented 6 years ago

Hi,

I'm using netty-tcnative-boringssl-static with IBM J9 JDK (websphere) for client cert auth to connect to Apple APNS (HTTP2), and getting this error (with System.setProperty("javax.net.debug", "all");):

ssl: Ignoring alias kes: signature does not conform to negotiated signature algorithms

Netty 4.1.30.Final
netty-tcnative-boringssl-static 2.0.18.Final

IBM JDK details:

java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 8.0.5.20 - pwa6480sr5fp20-20180802_01(SR5 FP20))
IBM J9 VM (build 2.9, JRE 1.8.0 Windows 10 amd64-64-Bit Compressed References 20180731_393394 (JIT enabled, AOT enabled)
OpenJ9   - bd23af8
OMR      - ca1411c
IBM      - 98805ca)
JCL - 20180719_01 based on Oracle jdk8u181-b12

This issue is seen both Windows & Linux. The same code is working with Oracle JDK 1.8. I tried to check if any algorithm is disabled as per this link, but none of it is working.

This same error is reported here also: https://groups.google.com/forum/#!topic/pushy-apns/CELffCBOfmk

Thanks.

normanmaurer commented 6 years ago

Can you provide a minimal reproducer I can run ?

Am 18.10.2018 um 17:58 schrieb Manikanta G notifications@github.com:

Hi,

I'm using netty-tcnative-boringssl-static with IBM J9 JDK (websphere) for client cert auth to connect to Apple APNS (HTTP2), and getting this error (with System.setProperty("javax.net.debug", "all");):

ssl: Ignoring alias kes: signature does not conform to negotiated signature algorithms Netty 4.1.30.Final

netty-tcnative-boringssl-static 2.0.18.Final

IBM JDK details:

java version "1.8.0_181" Java(TM) SE Runtime Environment (build 8.0.5.20 - pwa6480sr5fp20-20180802_01(SR5 FP20)) IBM J9 VM (build 2.9, JRE 1.8.0 Windows 10 amd64-64-Bit Compressed References 20180731_393394 (JIT enabled, AOT enabled) OpenJ9 - bd23af8 OMR - ca1411c IBM - 98805ca) JCL - 20180719_01 based on Oracle jdk8u181-b12 This issue is seen both Windows & Linux. The same code is working with Oracle JDK 1.8. I tried to check if any algorithm is disabled as per this link, but none of it is working.

This same error is reported here also: https://groups.google.com/forum/#!topic/pushy-apns/CELffCBOfmk

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

normanmaurer commented 6 years ago

@manikantag

manikantag commented 6 years ago

@normanmaurer, excuse me for the delay. I faced issue while extracting minimal repro code. Initially, I didn't added BouncyCastly dependency in the repro pom (it was there in our main project pom) and facing KeyManagerFactory not supported error). I had to debug to see the BouncyCastle requirement, especially for non Oracle JDKs.

Please find the repro code here: https://github.com/manikantag/netty-boringssl-test

You need to replace the APNS certificate path & password in com.manikanta.ApnsTest.test() method (any valid iOS development push certificate would do; if using prod cert, change the last arg to true). Please let me know if I should provide cert too (I'll share that over mail privately - ping me on Gitter @manikantag; I'm online now)

manikantag commented 6 years ago

I've downloaded IBM JDK 8.0.5.22 from here: https://developer.ibm.com/javasdk/downloads/sdk8/

manikantag commented 6 years ago

@normanmaurer did you get a chance to look into this issue?

normanmaurer commented 6 years ago

@manikantag sorry not yet and it may take me a few days as I am preparing for some business travel etc.

normanmaurer commented 6 years ago

@manikantag also I am confused... Bouncycastle is not required at all. Why you think it is ?

normanmaurer commented 6 years ago

Also I guess you will not be able to create a reproducer only using netty (without vertx) ? This would make it a lot easier .

manikantag commented 6 years ago

@manikantag also I am confused... Bouncycastle is not required at all. Why you think it is ?

Yes, BouncyCastle is being used for IBM JDK.

In OpenSsl.java#L165, cert = new SelfSignedCertificate() is checking for BouncyCastle in SelfSignedCertificate#L154 as it could not find sun.security.x509.X509CertImpl class in IBM JDK.

If I've not included BouncyCastle, then new SelfSignedCertificate() is causing an exception and control is going to OpenSsl.java#L181 and useKeyManagerFactory is never being set to true and thus getting KeyManagerFactory not supported error.

IBM security providers _(from ..\ibmsdk80\jre\lib\security\java.security):

security.provider.1=com.ibm.jsse2.IBMJSSEProvider2
security.provider.2=com.ibm.crypto.provider.IBMJCE
security.provider.3=com.ibm.security.jgss.IBMJGSSProvider
security.provider.4=com.ibm.security.cert.IBMCertPath
security.provider.5=com.ibm.security.sasl.IBMSASL
security.provider.6=com.ibm.xml.crypto.IBMXMLCryptoProvider
security.provider.7=com.ibm.xml.enc.IBMXMLEncProvider
security.provider.8=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
security.provider.9=sun.security.provider.Sun

Oracle JDK security providers (from ..\java1.8\jre8\lib\security\java.security):

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.mscapi.SunMSCAPI

manikantag commented 6 years ago

Also I guess you will not be able to create a reproducer only using netty (without vertx) ? This would make it a lot easier .

Yeah, I couldn't at this time. Learning netty is my next plan and I assuming your book on Netty is a good starting point :)

manikantag commented 6 years ago

@normanmaurer Till the time we have a fix for this, is there any better alternative way to support all OS versions (Windows, Linux & Oracle JDK, IBM JDK)?

I tried using netty-tcnative on Linux, but getting below error:

<dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative</artifactId>
            <version>2.0.18.Final</version>
            <classifier>linux-x86_64</classifier>
        </dependency>

Exception in thread "main" io.vertx.core.VertxException: OpenSSL is not available
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:74)
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144)
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133)
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298)
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106)
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70)
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46)
        at com.manikanta.Apns.main(Apns.java:33)
Caused by: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative_x86_64, netty_tcnative]
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:440)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73)
        ... 7 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_linux_x86_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_linux_x86_64.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_linux_x86_64_fedora
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_linux_x86_64_fedora.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64_fedora (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64_fedora (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_x86_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_x86_64.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_x86_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_x86_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative6660313035974227441.so (libssl.so.1.0.0: cannot open shared object file: No such file or directory)
                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                at java.lang.System.load(System.java:554)
                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:215)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative6660313035974227441.so (libssl.so.1.0.0: cannot open shared object file: No such file or directory)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.System.load(System.java:554)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                        at java.lang.reflect.Method.invoke(Method.java:508)
                        at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                        at java.security.AccessController.doPrivileged(AccessController.java:638)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                        ... 12 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more

But I can see the libnetty_tcnative.so in netty-tcnative-2.0.18.Final-linux-x86_64.jar\META-INF\native\libnetty_tcnative.so. I've installed OpenSSL and libapr too.

openssl version: OpenSSL 1.0.2k-fips 26 Jan 2017

apr: yum --showduplicates list apr | expand

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: centos.excellmedia.net
 * epel: del-repos.extreme-ix.org
 * extras: centos.excellmedia.net
 * updates: centos.excellmedia.net
Installed Packages
apr.x86_64                         1.4.8-3.el7_4.1                         @base
Available Packages
apr.i686                           1.4.8-3.el7_4.1                         base
apr.x86_64                         1.4.8-3.el7_4.1                         base

manikantag commented 6 years ago

I tried compiling netty-tcnative-openssl-static-{os_arch} for Linux CentOS, but getting but is asking for directories for APR & OpenSSL. Should I give OpenSSL & APR installed dirs?

normanmaurer commented 6 years ago

@manikantag sorry like I said I am a bit busy atm to fully investigate... You should be able to build it by just doing:

./mvnw -pl openssl-static clean package

That said if the boringssl does not work this will not work as well.

Also if you want to use netty-tcnative directly on centos you will need to use:

            <classifier>linux-x86_64-fedora</classifier>

normanmaurer commented 6 years ago

@manikantag can you please also share the cert stuff so I can reproduce it via email if you can not do here:

norman dot maurer at googlemail dot com

manikantag commented 6 years ago

@normanmaurer Shared the test cert & pass over email. Thanks

manikantag commented 6 years ago

I tried using netty-tcnative-2.0.18.Final-linux-x86_64-fedora, and the previous problem is gone. But, now getting different issue about OpenSSL version: ALPN is not available for OpenSSL SSL/TLS engine

Although my server has OpenSSL 1.0.2k-fips 26 Jan 2017, but netty is somehow picking OpenSSL 1.0.1e-fips 11 Feb 2013.

Just to verify, I've removed OpenSSL and verified openssl version and there are no openssl versions listed. But even then netty is picking up OpenSSL 1.0.1e-fips 11 Feb 2013, which I'm not able to understand. Then again installed using yum install openssl and verified the version. How will netty look for openssl?

[root@localhost test]# yum install openssl
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.fibergrid.in
 * epel: epel.mirror.angkasa.id
 * extras: mirrors.fibergrid.in
 * updates: mirrors.fibergrid.in
Resolving Dependencies
--> Running transaction check
---> Package openssl.x86_64 1:1.0.2k-12.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================================================================================
 Package                                           Arch                                             Version                                                    Repository                                      Size
====================================================================================================================================================================================================================
Installing:
 openssl                                           x86_64                                           1:1.0.2k-12.el7                                            base                                           492 k

Transaction Summary
====================================================================================================================================================================================================================
Install  1 Package

Total download size: 492 k
Installed size: 814 k
Is this ok [y/d/N]: y
Downloading packages:
openssl-1.0.2k-12.el7.x86_64.rpm                                                                                                                                                             | 492 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:openssl-1.0.2k-12.el7.x86_64                                                                                                                                                                   1/1
  Verifying  : 1:openssl-1.0.2k-12.el7.x86_64                                                                                                                                                                   1/1

Installed:
  openssl.x86_64 1:1.0.2k-12.el7

[root@localhost test]# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

But, netty debug log shows that it is loading OpenSSL 1.0.1e-fips 11 Feb 2013:

2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default'
2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013

Also, below log is seen even when BouncyCastle is in classpath (not sure if this is related, though):

2018-10-22 11:29:12.445 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported.

Full debug log:

[root@localhost test]# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

[root@localhost test]# /opt/IBM/WebSphere/AppServer/java/bin/java -jar netty-boringssl-test.jar DevCert.p12 xxx false
2018-10-22 11:22:18.814 +0530 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.targetRecords: 4
2018-10-22 11:22:18.868 +0530 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4
2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 8
2018-10-22 11:22:18.918 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
2018-10-22 11:22:18.920 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: available
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - maxDirectMemory: 536870912 bytes (maybe)
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2018-10-22 11:22:18.923 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2018-10-22 11:22:18.924 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: 536870912 bytes
2018-10-22 11:22:18.925 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.CleanerJava6 - java.nio.ByteBuffer.cleaner(): available
2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
2018-10-22 11:22:18.936 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
2018-10-22 11:22:19.229 +0530 [main] DEBUG io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider - Default DNS servers: [/10.10.19.11:53, /10.20.19.11:53] (sun.net.dns.ResolverConfiguration)
2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: true
2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
2018-10-22 11:22:19.238 +0530 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
2018-10-22 11:22:19.239 +0530 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128
2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
2018-10-22 11:22:19.312 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5ca73065
2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true
2018-10-22 11:22:19.909 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true
2018-10-22 11:22:19.910 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying other loading mechanism.
2018-10-22 11:22:19.911 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.912 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying next name...
2018-10-22 11:22:19.913 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying other loading mechanism.
2018-10-22 11:22:19.914 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.915 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying next name...
2018-10-22 11:22:19.916 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying other loading mechanism.
2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying next name...
2018-10-22 11:22:19.918 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying other loading mechanism.
2018-10-22 11:22:19.919 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.927 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Successfully loaded the library /tmp/libnetty_tcnative8672013229519390435.so
2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default'
2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-10-22 11:22:20.050 +0530 [main] DEBUG io.netty.handler.ssl.util.SelfSignedCertificate - Failed to generate a self-signed X.509 certificate using sun.security.x509:
java.lang.NoClassDefFoundError: sun.security.x509.X509CertImpl
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:149) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:110) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:88) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:79) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:165) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298) [vertx-core-3.5.4.jar:3.5.4]
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.main(Apns.java:33) [netty-boringssl-test.jar:?]
Caused by: java.lang.ClassNotFoundException: sun.security.x509.X509CertImpl
        at java.net.URLClassLoader.findClass(URLClassLoader.java:609) ~[?:1.8.0]
        at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:924) ~[?:2.9 (11-22-2017)]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:869) ~[?:2.9 (11-22-2017)]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329) ~[?:1.8.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:852) ~[?:2.9 (11-22-2017)]
        ... 13 more
2018-10-22 11:22:20.734 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2018-10-22 11:22:20.735 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 4
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 4
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: true
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2018-10-22 11:22:20.763 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@80163c3
2018-10-22 11:22:20.770 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxSharedCapacityFactor: 2
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.linkCapacity: 16
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
2018-10-22 11:22:20.780 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported.
2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
... similar logs for other alogs...
2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
2018-10-22 11:22:20.827 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_KRB5_WITH_RC4_128_MD5 => KRB5-RC4-MD5
2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Supported protocols (OpenSSL): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
Exception in thread "main" io.vertx.core.VertxException: ALPN is not available for OpenSSL SSL/TLS engine
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:91)
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144)
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133)
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298)
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106)
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70)
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46)
        at com.manikanta.Apns.main(Apns.java:33)

manikantag commented 6 years ago

@normanmaurer I've updated the repro program to remove Vert.x usage. Now the test program just uses Netty (I've used Netty's HttpClient example program).

Usage: /ibmjava_home/bin/java com.manikanta.netty.Http2Client cert-path.p12 cert-pass

If everything works fine, we should get {"reason":"MissingDeviceToken"} as the response. Instead, if we get {"reason":"MissingProviderToken"}, then client cert auth is not working.

You can use the DevCert.p12 & pass I've mailed.

Let me know if this helps.

normanmaurer commented 6 years ago

Can you show me the content of /use/lib/ and /usr/lib64 ?

Am 22.10.2018 um 08:10 schrieb Manikanta G notifications@github.com:

I tried using netty-tcnative-2.0.18.Final-linux-x86_64-fedora, and the previous problem is gone. But, now getting different issue about OpenSSL version: ALPN is not available for OpenSSL SSL/TLS engine

Although my server has OpenSSL 1.0.2k-fips 26 Jan 2017, but netty is somehow picking OpenSSL 1.0.1e-fips 11 Feb 2013.

Just to verify, I've removed OpenSSL and verified openssl version and there are no openssl versions listed. But even then netty is picking up OpenSSL 1.0.1e-fips 11 Feb 2013, which I'm not able to understand. Then again installed using yum install openssl and verified the version. How will netty look for openssl?

[root@localhost kmstest]# yum install openssl Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile

base: mirrors.fibergrid.in

epel: epel.mirror.angkasa.id

extras: mirrors.fibergrid.in

updates: mirrors.fibergrid.in Resolving Dependencies --> Running transaction check ---> Package openssl.x86_64 1:1.0.2k-12.el7 will be installed --> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================================================================================================== Package Arch Version Repository Size

Installing: openssl x86_64 1:1.0.2k-12.el7 base 492 k

Transaction Summary

Install 1 Package

Total download size: 492 k Installed size: 814 k Is this ok [y/d/N]: y Downloading packages: openssl-1.0.2k-12.el7.x86_64.rpm | 492 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:openssl-1.0.2k-12.el7.x86_64 1/1 Verifying : 1:openssl-1.0.2k-12.el7.x86_64 1/1

Installed: openssl.x86_64 1:1.0.2k-12.el7 [root@localhost test]# openssl version OpenSSL 1.0.2k-fips 26 Jan 2017 But, netty debug log shows that it is loading OpenSSL 1.0.1e-fips 11 Feb 2013:

2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default' 2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013 Also, below log is seen even when BouncyCastle is in classpath (not sure if this is related, though):

2018-10-22 11:29:12.445 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported. Full debug log:

[root@localhost test]# openssl version OpenSSL 1.0.2k-fips 26 Jan 2017

[root@localhost test]# /opt/IBM/WebSphere/AppServer/java/bin/java -jar netty-boringssl-test.jar DevCert.p12 xxx false 2018-10-22 11:22:18.814 +0530 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework 2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple 2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.targetRecords: 4 2018-10-22 11:22:18.868 +0530 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4 2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false 2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512 2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false 2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 8 2018-10-22 11:22:18.918 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available 2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available 2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available 2018-10-22 11:22:18.920 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: available 2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true 2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9 2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.(long, int): available 2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available 2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - maxDirectMemory: 536870912 bytes (maybe) 2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) 2018-10-22 11:22:18.923 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model) 2018-10-22 11:22:18.924 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: 536870912 bytes 2018-10-22 11:22:18.925 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1 2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.CleanerJava6 - java.nio.ByteBuffer.cleaner(): available 2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false 2018-10-22 11:22:18.936 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available 2018-10-22 11:22:19.229 +0530 [main] DEBUG io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider - Default DNS servers: [/10.10.19.11:53, /10.20.19.11:53] (sun.net.dns.ResolverConfiguration) 2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: true 2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false 2018-10-22 11:22:19.238 +0530 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) 2018-10-22 11:22:19.239 +0530 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128 2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true 2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true 2018-10-22 11:22:19.312 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5ca73065 2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir) 2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true 2018-10-22 11:22:19.909 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true 2018-10-22 11:22:19.910 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying other loading mechanism. 2018-10-22 11:22:19.911 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 2018-10-22 11:22:19.912 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying next name... 2018-10-22 11:22:19.913 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying other loading mechanism. 2018-10-22 11:22:19.914 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 2018-10-22 11:22:19.915 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying next name... 2018-10-22 11:22:19.916 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying other loading mechanism. 2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying next name... 2018-10-22 11:22:19.918 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying other loading mechanism. 2018-10-22 11:22:19.919 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp 2018-10-22 11:22:19.927 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Successfully loaded the library /tmp/libnetty_tcnative8672013229519390435.so 2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default' 2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013 2018-10-22 11:22:20.050 +0530 [main] DEBUG io.netty.handler.ssl.util.SelfSignedCertificate - Failed to generate a self-signed X.509 certificate using sun.security.x509: java.lang.NoClassDefFoundError: sun.security.x509.X509CertImpl at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:149) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:110) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:88) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:79) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.ssl.OpenSsl.(OpenSsl.java:165) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73) [vertx-core-3.5.4.jar:3.5.4] at io.vertx.core.net.impl.SSLHelper.(SSLHelper.java:144) [vertx-core-3.5.4.jar:3.5.4] at io.vertx.core.http.impl.HttpClientImpl.(HttpClientImpl.java:133) [vertx-core-3.5.4.jar:3.5.4] at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298) [vertx-core-3.5.4.jar:3.5.4] at com.manikanta.Apns.createHTTP2Client(Apns.java:106) [netty-boringssl-test.jar:?] at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70) [netty-boringssl-test.jar:?] at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46) [netty-boringssl-test.jar:?] at com.manikanta.Apns.main(Apns.java:33) [netty-boringssl-test.jar:?] Caused by: java.lang.ClassNotFoundException: sun.security.x509.X509CertImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:609) ~[?:1.8.0] at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:924) ~[?:2.9 (11-22-2017)] at java.lang.ClassLoader.loadClass(ClassLoader.java:869) ~[?:2.9 (11-22-2017)] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329) ~[?:1.8.0] at java.lang.ClassLoader.loadClass(ClassLoader.java:852) ~[?:2.9 (11-22-2017)] ... 13 more 2018-10-22 11:22:20.734 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 2018-10-22 11:22:20.735 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 4 2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 4 2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192 2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11 2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192 2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: true 2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled 2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0 2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384 2018-10-22 11:22:20.763 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@80163c3 2018-10-22 11:22:20.770 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096 2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxSharedCapacityFactor: 2 2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.linkCapacity: 16 2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8 2018-10-22 11:22:20.780 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported. 2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384 ... similar logs for other alogs... 2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384 2018-10-22 11:22:20.827 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_KRB5_WITH_RC4_128_MD5 => KRB5-RC4-MD5 2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Supported protocols (OpenSSL): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] 2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA] Exception in thread "main" io.vertx.core.VertxException: ALPN is not available for OpenSSL SSL/TLS engine at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:91) at io.vertx.core.net.impl.SSLHelper.(SSLHelper.java:144) at io.vertx.core.http.impl.HttpClientImpl.(HttpClientImpl.java:133) at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298) at com.manikanta.Apns.createHTTP2Client(Apns.java:106) at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70) at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46) at com.manikanta.Apns.main(Apns.java:33) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

manikantag commented 6 years ago

Please find the attachments.

[root@localhost ~]# ls /usr/lib > lib.txt: lib.txt

[root@localhost ~]# ls /usr/lib64 > lib64.txt: lib64.txt

CentOS distro details:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# rpm --query centos-release
centos-release-7-4.1708.el7.centos.x86_64

[root@localhost ~]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

[root@localhost ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Please let me know if you any more details.

manikantag commented 6 years ago

@normanmaurer is there anything I can do to stipulate the fix for this? Can you give me a clue where could be an issue? As long as it is in Java code, I could try it (I'm not much into C/C++)

normanmaurer commented 6 years ago

Sorry but I had no time yet and may not have till next week due Business Travel. So to make it short I have no idea yet

Am 24.10.2018 um 08:16 schrieb Manikanta G notifications@github.com:

@normanmaurer is there anything I can do to stipulate the fix for this? Can you give me a clue where could be an issue? As long as it is in Java code, I could try it (I'm not much into C/C++)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

manikantag commented 6 years ago

Update: I've compiled latest 4.1 code to see if any new PRs (esp https://github.com/netty/netty/pull/8415) fixes it.

Luckily it is working with OpenSSL static combination (same is not working with 4.1.30-Final artifacts).

But BoringSSL static still doesn't work with latest code too. I've compiled both OpenSSL static & BoringSSL static on master branch.

manikantag commented 6 years ago

I can see some differences in the way the Cipher suites got loaded (OpenSSL Cipher suite mapping has more entries in same IBM J9 JDK).

openssl debug log: openssl.txt

boringssl debug log: boringssl.txt

normanmaurer commented 6 years ago

@manikantag thats most likely due the fact that BoringSSL does not support some of the ciphers that OpenSSL does. Can you tell me what cipher you require and can you check if this is supported by BoringSSL ?

manikantag commented 6 years ago

But the same BoringSSL is working with Oracle JDK with same cert & server, and the cipher suites loaded are exactly same with both JDKs.

Please find the debug logs for both IBM & Oracle JDKs.

BoringSSL + Oracle JDK: boringssl-oraclejdk.txt Boring SSL + IBM JDK: boringssl-ibmjdk.txt

OpenSSL + Oracle JDK: openssl-oraclejdk.txt OpenSSL + IBM JDK: openssl-ibmjdk.txt

Stack:

Netty 4.1.31.Final-SNAPSHOT
netty-tcnative-boringssl-static-2.0.20.Final-SNAPSHOT-linux-x86_64.jar
netty-tcnative-openssl-static-2.0.19.Final-SNAPSHOT-linux-x86_64.jar

Below additional exception is coming with IBM JDK for both OpenSSL & BoringSSL:

2018-10-25 14:21:21.246 +0530 [main] DEBUG io.netty.handler.ssl.OpenSslX509TrustManagerWrapper - Unable to access wrapped TrustManager
java.lang.NoSuchFieldException: contextSpi

I've verified IBM JDK javax.net.ssl.SSLContext (from ibmjssefw.jar) and the class doesn't have contextSpi field. Not sure if this related to this issue.

manikantag commented 6 years ago

I'm trying to see what exact algo is causing this, but so far no luck. I'll be working on that.

normanmaurer commented 6 years ago

@manikantag any update ? I hope to be able to also have a look very soon.

normanmaurer commented 6 years ago

@manikantag also did you see https://www-01.ibm.com/support/docview.wss?uid=swg22003076 ?

manikantag commented 6 years ago

@manikantag also did you see https://www-01.ibm.com/support/docview.wss?uid=swg22003076 ?

Yes. In fact I've removed all the algos in the ibm_sdk80\jre\lib\security\java.security file (shown below)

jdk.certpath.disabledAlgorithms=MD2
jdk.tls.disabledAlgorithms=SSLv3
jdk.jar.disabledAlgorithms=MD2

Removed everything else. But still the same error is coming

And I can confirm that without removing any algos from ibmjdk' java.security, client cert auth is working fine for APNS TCP connection (previous way to send iOS push) using same certificate. Also, as I mentioned the same BoringSSL is working with Oracle JDK with same cert & server, and the cipher suites loaded are exactly same with both Oracle & IBM JDKs.

manikantag commented 6 years ago

@manikantag any update ? I hope to be able to also have a look very soon.

I'm not able to figure out which algo is causing this issue, and in my observation as stated in my previous comment, cipher suites are same with both JDKs.

manikantag commented 6 years ago

Both Oracle & IBM have same jdk.certpath.disabledAlgorithms & jdk.jar.disabledAlgorithms.

jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024

But jdk.tls.disabledAlgorithms differ: DESede is not there in Oracle JDK. I tried removing that from IBM JDK, but didn't worked.

IBM:

jdk.tls.disabledAlgorithms=SSLv3,  TLS1, TLS1.1, RC4, MD5withRSA, DH keySize < 1024, 3DES_EDE_CBC, DESede, \
    EC keySize < 224, DES40_CBC, RC4_40

Oracle:

jdk.tls.disabledAlgorithms=SSLv3, TLS1, TLS1.1, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC

normanmaurer commented 6 years ago

@manikantag sorry but I think I will not be able to help here... I have no license for IBM JDK and I can not reproduce it with OpenJ9 (java11) . Maybe if you can send me the log when it fails I get a better idea.

manikantag commented 6 years ago

@normanmaurer I didn't get what do you mean by IBM JDK license? I've downloaded IBM JDK 8.0.5.22 from here: https://developer.ibm.com/javasdk/downloads/sdk8/ and just unzipped the jdk and it is working fine. In fact I don't have any license either.

I've attached Netty debug logs for different combinations in my previous comment: https://github.com/netty/netty-tcnative/issues/404#issuecomment-433315655

Please let me know which log in specific you needed.

manikantag commented 6 years ago

@normanmaurer If needed I can provide you screen sharing and remote debug that way. Please let me know your preference.

normanmaurer commented 6 years ago

@manikantag this one http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?popup=Y&la_formnum=&li_formnum=L-SMKR-AVSEUH&title=IBM%20SDK,%20Java%20Technology%20Edition,%20Version%208&accepted_url=http://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java/8.0.5.25/linux/i386/ibm-java-sdk-8.0-5.25-i386-archive.bin ?

manikantag commented 6 years ago

@normanmaurer The moment I click 'I agree', ibm-java-sdk-8.0-5.25-i386-archive.bin is downloading. Isn't that the JDK bin? Am I missing something?

manikantag commented 6 years ago

@normanmaurer I've downloaded the JRE from the link you've given and able to install it in CentOS. It didn't asked me any license file.

normanmaurer commented 6 years ago

@manikantag so after some more debugging I can say that the KeyManagerFactory that is provided by the IBM JDK does return no alias when calling chooseEngineClientAlias(...) while the JDK one does. The problem here really is that I have no idea why as I can not see the implementation of the IBM JDK. So I am not sure how I should debug this any further. I can only say that the same call against OpenJDK / OracleJDK produces and alias.

manikantag commented 6 years ago

@normanmaurer I tried to debug and struck at IBM JDK's chooseEngineClientAlias() as I'm not able to decompile that method.

In another trail, I changed the io.netty.handler.ssl.OpenSslKeyMaterialManager.chooseClientAlias() implementation like below:

private String chooseClientAlias(ReferenceCountedOpenSslEngine engine,
                                 String[] keyTypes, X500Principal[] issuer) {
    X509KeyManager manager = provider.keyManager();
    if (manager instanceof X509ExtendedKeyManager) {
        String alias = ((X509ExtendedKeyManager) manager).chooseEngineClientAlias(keyTypes, issuer, engine);
        if (alias != null) { // ------> return if not null. Otherwise let 'manager.chooseClientAlias()' execute
            return alias;
        }
    }
    return manager.chooseClientAlias(keyTypes, issuer, null);
}

I'm returning the chooseEngineClientAlias() response only if it is not null, which will be case with Oracle JDK. In case of IBM, manager.chooseClientAlias() will be called due to the null check.

This solved my issue. Not sure about any consequences, though. Can you please confirm if we can let the code go till return manager.chooseClientAlias(keyTypes, issuer, null) even if KeyManager instanceof X509ExtendedKeyManager = true? (From Git history, I can see it is like this before the current change - https://github.com/netty/netty/commit/0337ecdcc87ae19b9f329eeffefa336354c1d0a0#diff-5a3c25c75e8b9e3df59424c534f36993L173)

But I also have another doubt: How come the same code with same IBM JDK is working fine with OpenSSL static instead of BoringSSL? When using netty-tcnative-openssl-static-2.0.19.Final-SNAPSHOT-linux-x8664.jar (which I've compiled from source)_, chooseEngineClientAlias(keyTypes, issuer, engine) itself is returning the correct alias with both Oracle & IBM JDKs. But when using BoringSSL, same line is returning null with IBM JDK.

Below is the summary:

JDK	Native SSL type	chooseEngineClientAlias()
Oracle	BoringSSL	alias string
Oracle	OpenSSL	alias string
IBM	BoringSSL	null
IBM	OpenSSL	alias string

normanmaurer commented 6 years ago

@manikantag sure you could do the change but this is really just a workaround and not 100 % correct imho. That said for this you don't need to adjust Netty at all but you could just wrap the KeyManagerFactory (and so the X509ExtendedKeyManager to do this.

Its interesting that OpenSSL works, I suspect it uses different signature algorithms. I will check soon and verify if this is the case.

manikantag commented 6 years ago

@normanmaurer I didn't get what do you mean by

you could just wrap the KeyManagerFactory (and so the X509ExtendedKeyManager to do this.

Does that mean a custom implementation of KeyManagerFactory and X509ExtendedKeyManager to select the alias correctly?

normanmaurer commented 6 years ago

@manikantag yes... you can basically just wrap the original one and do the extra work.

normanmaurer commented 6 years ago

@manikantag good news... I think I found the problem and working on a fix as we speak now.

normanmaurer commented 6 years ago

@manikantag alright I figured it out... Once these two PRS are in all works:

https://github.com/netty/netty-tcnative/pull/412 https://github.com/netty/netty/pull/8481

Thanks again for all the details :)

manikantag commented 6 years ago

@normanmaurer great news, thanks for your efforts. Will check after PR merges.

manikantag commented 6 years ago

@normanmaurer Now that the 2 PRs are merged into 4.1 branch, when can we expect the 4.1.32 release? Thanks.

normanmaurer commented 6 years ago

@manikantag not sure yet... I would guess in the next two weeks.

tigersaggy commented 5 years ago

Thanks a lot for sharing this post, i was facing the same issue and was struggling with it for the last two days. After reading the post and upgrading the netty to version 4.1.32, it worked like a charm.

Kudos @manikantag and @normanmaurer :)

netty / netty-tcnative

Client certificate signature issue when using netty-tcnative-boringssl-static with IBM J9 JDK #404

==================================================================================================================================================================================================================== Package Arch Version Repository Size

Transaction Summary