sofastack / sofa-rpc

SOFARPC is a high-performance, high-extensibility, production-level Java RPC framework.
https://www.sofastack.tech/sofa-rpc/docs/Home
Apache License 2.0
3.84k stars 1.18k forks source link

协议H2,版本5.7.0,Server deserialize error,具体堆栈错误如下,之前协议时bolt时没有错误 #915

Closed xiaoming2624 closed 3 years ago

xiaoming2624 commented 4 years ago

com.alipay.sofa.rpc.core.exception.SofaRpcException: com.alipay.sofa.rpc.core.exception.SofaRpcException: Server deserialize error, readObject: unknown code 0x59 Y at com.alipay.sofa.rpc.transport.http.AbstractHttp2ClientTransport.syncSend(AbstractHttp2ClientTransport.java:258) at com.alipay.sofa.rpc.client.AbstractCluster.doSendMsg(AbstractCluster.java:535) at com.alipay.sofa.rpc.client.AbstractCluster.sendMsg(AbstractCluster.java:506) at com.alipay.sofa.rpc.filter.ConsumerInvoker.invoke(ConsumerInvoker.java:60) at com.alipay.sofa.rpc.filter.sofatracer.ConsumerTracerFilter.invoke(ConsumerTracerFilter.java:66) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98) at com.alipay.sofa.rpc.filter.RpcReferenceContextFilter.invoke(RpcReferenceContextFilter.java:80) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98) at com.alipay.sofa.rpc.filter.ConsumerExceptionFilter.invoke(ConsumerExceptionFilter.java:37) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98) at com.alipay.sofa.rpc.filter.FilterChain.invoke(FilterChain.java:262) at com.alipay.sofa.rpc.client.AbstractCluster.filterChain(AbstractCluster.java:499) at com.alipay.sofa.rpc.client.FailoverCluster.doInvoke(FailoverCluster.java:67) at com.alipay.sofa.rpc.client.AbstractCluster.invoke(AbstractCluster.java:288) at com.alipay.sofa.rpc.client.ClientProxyInvoker.invoke(ClientProxyInvoker.java:83) at com.jinninghui.newspiral.gateway.ServiceForPeer_proxy_1.addGenericMsg(ServiceForPeer_proxy_1.java) at com.jinninghui.newspiral.p2p.impl.hotstuff.ServiceForPeerClient.addGenericMsg(ServiceForPeerClient.java:187) at com.jinninghui.newspiral.p2p.impl.hotstuff.ChannelClientMgr.broadcastGenericMsg(ChannelClientMgr.java:105) at com.jinninghui.newspiral.p2p.impl.hotstuff.HotStuffRPC.broadcastGenericMsg(HotStuffRPC.java:29) at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.alipay.sofa.runtime.service.binding.JvmBindingAdapter$JvmServiceInvoker.doInvoke(JvmBindingAdapter.java:200) at com.alipay.sofa.runtime.spi.service.ServiceProxy.invoke(ServiceProxy.java:39) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy99.broadcastGenericMsg(Unknown Source) at com.jinninghui.newspiral.consensus.impl.hotstuff.NewSpiralHotStuffProcessor.tryCreateBlockAndBroadcast(NewSpiralHotStuffProcessor.java:624) at com.jinninghui.newspiral.consensus.impl.hotstuff.NewSpiralHotStuffProcessor.runOnceAsConsensusPeer(NewSpiralHotStuffProcessor.java:183) at com.jinninghui.newspiral.consensus.impl.hotstuff.NewSpiralHotStuffProcessor.run(NewSpiralHotStuffProcessor.java:98) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: com.alipay.sofa.rpc.core.exception.SofaRpcException: Server deserialize error, readObject: unknown code 0x59 Y at com.alipay.sofa.rpc.message.http.HttpResponseFuture.getNowResponse(HttpResponseFuture.java:158) at com.alipay.sofa.rpc.message.http.HttpResponseFuture.getSofaResponse(HttpResponseFuture.java:146) at com.alipay.sofa.rpc.transport.http.AbstractHttp2ClientTransport.doInvokeSync(AbstractHttp2ClientTransport.java:283) at com.alipay.sofa.rpc.transport.http.AbstractHttp2ClientTransport.syncSend(AbstractHttp2ClientTransport.java:252) ... 30 more Caused by: com.alipay.sofa.rpc.core.exception.SofaRpcException: Server deserialize error, readObject: unknown code 0x59 Y at com.alipay.sofa.rpc.transport.http.AbstractHttpClientHandler.receiveHttpResponse(AbstractHttpClientHandler.java:173) at com.alipay.sofa.rpc.transport.http.Http2ClientChannelHandler.channelRead0(Http2ClientChannelHandler.java:99) at com.alipay.sofa.rpc.transport.http.Http2ClientChannelHandler.channelRead0(Http2ClientChannelHandler.java:46) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.http2.InboundHttp2ToHttpAdapter.fireChannelRead(InboundHttp2ToHttpAdapter.java:138) at io.netty.handler.codec.http2.InboundHttp2ToHttpAdapter.onDataRead(InboundHttp2ToHttpAdapter.java:249) at io.netty.handler.codec.http2.DelegatingDecompressorFrameListener.onDataRead(DelegatingDecompressorFrameListener.java:78) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:236) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118) at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1429) at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1211) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1245) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ... 1 more

OrezzerO commented 4 years ago

More information need to solve this problem. Can you provide a simple demo for us to debug?

xiaoming2624 commented 4 years ago

@OrezzerO 您好,十分感谢,我写了一个demo在网盘里,具体地址如下 链接: https://pan.baidu.com/s/1f103BcHS1bkEqX-AWB1uWg 提取码: pppn

具体访问报错如下

image

image

堆栈信息如下, com.alipay.sofa.rpc.core.exception.SofaRpcException: Server deserialize error, readObject: unknown code 0x32 2 at com.alipay.sofa.rpc.transport.http.AbstractHttpClientHandler.receiveHttpResponse(AbstractHttpClientHandler.java:173) ~[sofa-rpc-all-5.7.0.jar:5.7.0] at com.alipay.sofa.rpc.transport.http.Http2ClientChannelHandler.channelRead0(Http2ClientChannelHandler.java:100) ~[sofa-rpc-all-5.7.0.jar:5.7.0] at com.alipay.sofa.rpc.transport.http.Http2ClientChannelHandler.channelRead0(Http2ClientChannelHandler.java:47) ~[sofa-rpc-all-5.7.0.jar:5.7.0] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.InboundHttp2ToHttpAdapter.fireChannelRead(InboundHttp2ToHttpAdapter.java:136) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.InboundHttp2ToHttpAdapter.onDataRead(InboundHttp2ToHttpAdapter.java:247) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DelegatingDecompressorFrameListener.onDataRead(DelegatingDecompressorFrameListener.java:79) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:292) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-all-4.1.45.Final.jar:4.1.45.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

xiaoming2624 commented 4 years ago

hi, @OrezzerO ,the sharing link above is about to expire, Please look at the following :

testDemo.zip

xiaoming2624 commented 4 years ago

hi, @OrezzerO,I hop your timely feedback,think you

OrezzerO commented 4 years ago

Sorry for that. I have downloaded it and have reproduce the issue. In some case, it will occur problem when payload is large. I will focus on it this week.

xiaoming2624 commented 4 years ago

@OrezzerO hello, thank you very much I quote your modified (Http2ServerChannelHandler.java)

Protocol:h2 sofa-rpc-all:version: 5.6.5

report errors in Slf4JLogger,but function is not affected temporarily

ERROR as follows: 2020-06-03 10:51:50,841 ERROR io.netty.util.internal.logging.Slf4JLogger.error:171 LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:171) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:105) com.alipay.sofa.rpc.transport.http.Http2ServerChannelHandler.onDataRead(Http2ServerChannelHandler.java:122) io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:236) io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421) io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118) io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390) io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

OrezzerO commented 4 years ago

@xiaoming2624 Thank you for your report. I'll check if there is a memory leak problem.

xiaoming2624 commented 4 years ago

@OrezzerO Thank you very much. It should be a must When I start up works.