[gateway] Unable to configure https endpoint with debian stable openjdk

vcariven commented 5 years ago

When configuring an https proxy endpoint target, the gateway fails with a java.lang.NoSuchMethodError exception when running with openjdk 8u181 from debian stable channel (stretch).

The error does not happen when installing openjdk 8u191 from unstable channel (sid).

It seems that sun.security.ssl.SSLSessionImpl.<init>() contructor implementation has changed between versions 8u181 and 8u191 which could be the error cause. I have noticed that no gravitee package is involved in this stacktrace but only netty ones. This surely means that this is related with embedded netty libraries.

Similar incompatibility issues regarding this class implementation have been reported on internet (either from 8u181 to 8u191 or the other way : 8u191 to 8u181) :

Expected Behavior

The proxified https endpoint should be made available through gravitee gateway without error with one or the other jvm (8u181 or 8u191).

Current Behavior

The gateway reports the following error :

[2019-01-25 08:48:03,618][WARN][io.netty.channel.DefaultChannelPipeline][vert.x-eventloop-thread-1] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.NoSuchMethodError: sun.security.ssl.SSLSessionImpl.<init>(Lsun/security/ssl/ProtocolVersion;Lsun/security/ssl/CipherSuite;Ljava/util/Collection;Lsun/security/ssl/SessionId;Ljava/lang/String;IZ)V
    at sun.security.ssl.ClientHandshaker.serverHello(ClientHandshaker.java:723)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:210)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1408)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1316)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:255)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

Possible Solution

I tried to upgrade to openjdk 8u191 from unstable channel, but this is not a satifactory solution to install a package from an unstable channel.
I tried to run gravitee with the debian openjdk-11 (backport channel) but the gateway fails to start with the following startings logs : (no idea if gravitee is known to be compatible or not with jdk 11)

java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Debian-2bpo91)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Debian-2bpo91, mixed mode, sharing)

=========================================================================

  Gravitee.IO Standalone Runtime Bootstrap Environment

  GRAVITEE_HOME: /opt/graviteeio-gateway-1.21.2

  GRAVITEE_OPTS: 

  JAVA: /docker-java-home/bin/java

  JAVA_OPTS:  -javaagent:/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/graviteeio-gateway/config/gravitee-jmx-exporter.yml -Djava.net.preferIPv4Stack=true -Xms128m -Xmx256m -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8

  CLASSPATH: /opt/graviteeio-gateway-1.21.2/lib/gravitee-gateway-standalone-bootstrap-1.21.2.jar

=========================================================================

[jetty-alpn-agent] Could not find a matching alpn-boot JAR for Java version: 11.0.1
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/opt/graviteeio-gateway-1.21.2/lib/ext/spring-core-5.0.10.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2019-01-25 13:29:30,798][ERROR][io.gravitee.plugin.core.internal.PluginRegistryImpl][graviteeio-node] No plugin registry configured.
[2019-01-25 13:29:30,802][ERROR][io.gravitee.gateway.standalone.node.GatewayNode][graviteeio-node] An error occurs while starting component PluginRegistry
java.lang.RuntimeException: No plugin registry configured.
    at io.gravitee.plugin.core.internal.PluginRegistryImpl.init(PluginRegistryImpl.java:92)
    at io.gravitee.plugin.core.internal.PluginRegistryImpl.doStart(PluginRegistryImpl.java:81)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at io.gravitee.node.container.AbstractNode.doStart(AbstractNode.java:74)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at io.gravitee.node.container.AbstractContainer.doStart(AbstractContainer.java:89)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.gravitee.gateway.standalone.boostrap.Bootstrap.start(Bootstrap.java:162)
    at io.gravitee.gateway.standalone.boostrap.Bootstrap.main(Bootstrap.java:171)
[2019-01-25 13:29:30,804][ERROR][io.gravitee.gateway.standalone.GatewayContainer][graviteeio-node] An unexpected error occurs while starting Gravitee.io - API Gateway
java.lang.RuntimeException: No plugin registry configured.
    at io.gravitee.plugin.core.internal.PluginRegistryImpl.init(PluginRegistryImpl.java:92)
    at io.gravitee.plugin.core.internal.PluginRegistryImpl.doStart(PluginRegistryImpl.java:81)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at io.gravitee.node.container.AbstractNode.doStart(AbstractNode.java:74)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at io.gravitee.node.container.AbstractContainer.doStart(AbstractContainer.java:89)
    at io.gravitee.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:32)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.gravitee.gateway.standalone.boostrap.Bootstrap.start(Bootstrap.java:162)
    at io.gravitee.gateway.standalone.boostrap.Bootstrap.main(Bootstrap.java:171)
[17.962s][warning][jfr] Could not access .jfc-files in /usr/lib/jvm/java-11-openjdk-amd64/lib/jfr, /usr/lib/jvm/java-11-openjdk-amd64/lib/jfr

Steps to Reproduce (for bugs)

Run gravitee gateway with stable debian openjdk-8 (8u181)
Configure an https proxy endpoint
Call the proxified endpoint through gravitee gateway
The call fails and the gateway report the NoSuchMethodError

Context

Your Environment

Version used: 1.21.2
Browser Name and version: curl

Operating System and version: debian 9.6 (stretch) in docker container


java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

brasseld commented 5 years ago

Hi @vcariven

Please downgrade the ALPN agent from 2.0.9 to 2.0.8 api_gateway_home/boot https://search.maven.org/remotecontent?filepath=org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.8/jetty-alpn-agent-2.0.8.jar

Don't forget to update the bin/gravitee script to take care of this change.

Regards,

brasseld commented 5 years ago

I've miss to inform you that we are not supporting JDK 11 yet.

Regards,

vcariven commented 5 years ago

Hello,

Thanks for your quick answer and the information about the JDK 11 support. As it is not yet supported, JDK is though not an option. Have you got some explaination regarding the error reported and the behaviour difference between the two openjdk 8 minor versions (181 and 191) ?

brasseld commented 5 years ago

Hi @vcariven

Did you have a look to my previous comment ? https://github.com/gravitee-io/issues/issues/1854#issuecomment-457605014

vcariven commented 5 years ago

Sorry, I thought your comment was related to jdk11 (as startup logs in jdk11 mentioned alpn). I'll give the alpn downgrade a try today on jdk 8 and will keep you informed.

Thanks.

vcariven commented 5 years ago

Hi @brasseld

I have just tested with jetty-alpn-agent-2.0.8.jar (and the related changes in the boot script) and the same error is raised :

[2019-01-28 09:41:54,913][WARN][io.netty.channel.DefaultChannelPipeline][vert.x-eventloop-thread-4] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.NoSuchMethodError: sun.security.ssl.SSLSessionImpl.<init>(Lsun/security/ssl/ProtocolVersion;Lsun/security/ssl/CipherSuite;Ljava/util/Collection;Lsun/security/ssl/SessionId;Ljava/lang/String;IZ)V
    at sun.security.ssl.ClientHandshaker.serverHello(ClientHandshaker.java:723)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:210)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1408)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1316)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:255)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

brasseld commented 5 years ago

So you've tested 2.0.8 with which version of JDK ?

vcariven commented 5 years ago

openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

NicolasGeraud commented 5 years ago

hi @vcariven ,

I've tested gravitee 1.21.2 (with the default alpn.jar v2.0.9), with jdk 1.8.0_181 with success :

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.181-b13, mixed mode)

The bootstrap of the gateway :

=========================================================================

  Gravitee.IO Standalone Runtime Bootstrap Environment

  GRAVITEE_HOME: /Users/nicolasgeraud/Work/graviteeio-versions/graviteeio-full-1.21.2/graviteeio-gateway-1.21.2

  GRAVITEE_OPTS:

  JAVA: /Users/nicolasgeraud/.jenv/versions/1.8.0.181/bin/java

  JAVA_OPTS:  -Xms512m -Xmx512m -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8

  CLASSPATH: /Users/nicolasgeraud/Work/graviteeio-versions/graviteeio-full-1.21.2/graviteeio-gateway-1.21.2/lib/gravitee-gateway-standalone-bootstrap-1.21.2.jar

=========================================================================

Can you share a complete gravitee.log file ?

brasseld commented 5 years ago

Hi @vcariven

What is the status of this ticket ?

Thanks for your feedback.

Regards,

vcariven commented 5 years ago

Hello,

This seems to be related to an incompatibility between debian 9 (stretch) openjdk 8 package and netty library. I think @NicolasGeraud has reproduced the issue.

Regards

brasseld commented 5 years ago

No he doesn't:

I've tested gravitee 1.21.2 (with the default alpn.jar v2.0.9), with jdk 1.8.0_181 with success :

vcariven commented 5 years ago

I thought he had. Issue can be reproduced by building a gravitee docker image inheriting from docker.io/openjdk:8-jre-stretch

no-response[bot] commented 5 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

gravitee-io / issues