dapengzhang0
commented
2 years ago @donnadionne This is duplicated to https://github.com/grpc/grpc-java/issues/8880 which is being triage there. Please close the issue here. Thanks.
Closed WPCode closed 2 years ago
dapengzhang0
commented
2 years ago @donnadionne This is duplicated to https://github.com/grpc/grpc-java/issues/8880 which is being triage there. Please close the issue here. Thanks.
macche80
commented
2 years ago I am having the same issue actually (I was about to open an issue on this, when I saw that @WPCode already opened one).
I have a similar feedback:
This is the stack trace on the server side (which doesn't seem to have any problem), in case gives you any clue:
io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:167) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:356) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1000) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:956) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:512) at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:152) at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:209) at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:417) at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:617) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)
Code wise, I was also looking at DefaultHttp2ConnectionDecoder.java, line 339 and I didn't notice anything obvious, except connection being null, but then, as you said, it should not even be possible, because it's final, and it was used few lines above.
BUT I think what is happening is this:
boolean isInformational = !connection.isServer() && ...
connection.isServer() is returning null, and !null is throwing NPE.
EDIT: actually scratch that, because isServer is returning a primitive boolean, not Boolean, so no NPE from unboxing.
We are also using grpc 1.37.0 and netty 4.1.52.Final
WPCode
commented
2 years ago Closing as duplicate of https://github.com/grpc/grpc-java/issues/8880
What version of gRPC and what language are you using?
gRPC version: 1.37.0, we are using Java. The underlying netty version should be 4.1.52.
What operating system (Linux, Windows,...) and version?
Linux, inside a container inside kubernetes pod
What runtime / compiler are you using (e.g. python version or version of gcc)
Java version
What did you do?
We don't know how to reproduce this issue, but it starts showing up in several production servers. The gRPC calls one component make to other components using those BlockingStub will fail, and we see the NullPointerException below every time. If we restart the component (pod), it starts working again. Usually it work fine for some time (maybe several days or weeks?), then starts failing. Once it starts failing, it will fail forever. And we don't know how long it will take to fail again.
What did you expect to see?
gRPC calls work fine.
What did you see instead?
Anything else we should know about your project / environment?
We tried to look at DefaultHttp2ConnectionDecoder.java, line 339, it's weird this line will throw NPE, since connection is used before and java final, how could it throw NPE?