search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.82k stars 6.52k forks source link

[Bug] Skywalking + SpringCloud Gateway : Can not do async finish for the span repeatedly #7658

Closed Hlenn closed 2 years ago

Hlenn commented 3 years ago

Search before asking

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

I use docker compose to install skywalking Java agent

The following error occurred:,

I checked it at the same time https://github.com/apache/skywalking/issues/5742 The problem is quite,

Indicates that it was fixed in 8.2

However, when I use 8.5.0, 8.6.0 and 8.7.0, the following same error messages still appear.

2021-09-06 10:52:32.862 ERROR 1 --- [reactor-http-epoll-4] reactor.core.publisher.Operators : Operator called default onErrorDropped

java.lang.RuntimeException: Can not do async finish for the span repeatedly. at org.apache.skywalking.apm.agent.core.context.trace.AbstractTracingSpan.asyncFinish(AbstractTracingSpan.java:332) ~[skywalking-agent.jar:8.6.0] at org.apache.skywalking.apm.plugin.spring.webflux.v5.DispatcherHandlerHandleMethodInterceptor.lambda$afterMethod$0(DispatcherHandlerHandleMethodInterceptor.java:96) ~[na:na] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:156) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:139) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onComplete(MonoFlatMap.java:174) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:144) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.ignoreDone(MonoIgnoreThen.java:190) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreInner.onComplete(MonoIgnoreThen.java:240) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.Operators$MonoSubscriber.onComplete(Operators.java:1824) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onComplete(MonoIgnoreThen.java:314) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.Operators.complete(Operators.java:135) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.ignoreDone(MonoIgnoreThen.java:190) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreInner.onComplete(MonoIgnoreThen.java:240) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2016) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:129) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at rx.internal.util.ActionSubscriber.onCompleted(ActionSubscriber.java:49) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.SafeSubscriber.onCompleted(SafeSubscriber.java:79) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4$1.onCompleted(OperatorOnErrorResumeNextViaFunction.java:126) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onCompleted(OperatorOnErrorResumeNextViaFunction.java:101) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onCompleted(AbstractCommand.java:1466) ~[hystrix-core-1.5.18.jar!/:1.5.18] at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onCompleted(AbstractCommand.java:1391) ~[hystrix-core-1.5.18.jar!/:1.5.18] at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70) ~[rxjava-1.3.8.jar!/:1.3.8] at rx.internal.reactivestreams.SubscriberAdapter.onComplete(SubscriberAdapter.java:64) ~[rxjava-reactive-streams-1.2.1.jar!/:1.2.1] at reactor.core.publisher.StrictSubscriber.onComplete(StrictSubscriber.java:123) ~[reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2016) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onComplete(FluxOnAssembly.java:395) ~[reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:191) ~[reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onComplete(FluxContextStart.java:115) ~[reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260) [reactor-core-3.3.10.RELEASE.jar!/:3.3.10.RELEASE] at reactor.netty.FutureMono$FutureSubscription.operationComplete(FutureMono.java:180) ~[reactor-netty-0.9.12.RELEASE.jar!/:0.9.12.RELEASE] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:363) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:471) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:429) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:519) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:127) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) ~[netty-transport-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.52.Final-linux-x86_64.jar!/:4.1.52.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.52.Final.jar!/:4.1.52.Final] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]

What you expected to happen

My same Java program can run normally under docker. Once the agent is used, it will not work normally.

How to reproduce

Please give me some suggestions on how to solve it

Anything else

No response

Are you willing to submit PR?

Code of Conduct

wu-sheng commented 3 years ago

Do you locate what happens? (you clicked es I am willing to submit a PR!). Our gateway test cases about Gateway 3.x are here, https://github.com/apache/skywalking-java/tree/main/test/plugin/scenarios/gateway-3.x-scenario

You are better to do cross check.

Hlenn commented 3 years ago

感谢回复 但是我看了这个链接的内容后，还是不明白应该如何解决我现在遇到的问题。 能告诉我，我现在应该如何操作吗？

Hlenn commented 3 years ago

My gateway version is 2.2.5

I have opened the option of optional plug-ins in the config file.

wu-sheng commented 3 years ago

Plugins contributed by the community. We don't know how to deal with it. My point to give you the test demo is, it doesn't have issue, and all traces are reported normally. If you want to debug the error of a plugin, remote debug could help, and you need to understand how Gateway works in the source codes level. Which is also why we can't tell what is wrong, because we don't know.

wallezhang commented 3 years ago

@Hlenn Could you provide a project sample code that can reproduce this error?

wu-sheng commented 2 years ago

No update in 2 months.