search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.82k stars 6.52k forks source link

Skywalking + SpringCloud Gateway : Can not do async finish for the span repeatedly. #5742

Closed HeXsheng closed 4 years ago

HeXsheng commented 4 years ago

I used Skywalking in SpringCloud gateway, when testing a large number of users concurrently requesting the Gateway, I found this exception: "Can not do async finish for the span repeatedly."

version

Skywalking : 8.1.0 SpringCloudGateway: 2.0.4.RELEASE

Detail

2020-10-28 12:25:11,874 [reactor-http-epoll-3] ERROR [] [ContextHandler] - Error cannot be forwarded to user-facing Mono reactor.core.Exceptions$BubblingException: java.lang.RuntimeException: Can not do async finish for the span repeatedly. at reactor.core.Exceptions.bubble(Exceptions.java:154) at reactor.core.publisher.Operators.onErrorDropped(Operators.java:263) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:193) at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:201) at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:146) at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:87) at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:608) at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:138) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Can not do async finish for the span repeatedly. at org.apache.skywalking.apm.agent.core.context.trace.AbstractTracingSpan.asyncFinish(AbstractTracingSpan.java:314) at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:95) at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:83) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:189) ... 27 common frames omitted 2020-10-28 12:30:29,127 [reactor-http-epoll-1] ERROR [] [Operators] - Operator called default onErrorDropped java.lang.NullPointerException: null at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:87) at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:83) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:189) at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:201) at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:146) at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:87) at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:608) at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:138) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) at java.lang.Thread.run(Thread.java:748) 2020-10-28 12:30:29,127 [reactor-http-epoll-1] ERROR [] [ContextHandler] - Error cannot be forwarded to user-facing Mono reactor.core.Exceptions$BubblingException: java.lang.NullPointerException at reactor.core.Exceptions.bubble(Exceptions.java:154) at reactor.core.publisher.Operators.onErrorDropped(Operators.java:263) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:193) at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:201) at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:146) at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:87) at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:608) at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:138) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:87) at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:83) at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:189) ... 27 common frames omitted

wu-sheng commented 4 years ago

Do you recheck the 8.2.0? I remember there eas a fix about this.