search

spring-cloud / spring-cloud-sleuth

Distributed tracing for spring cloud
https://spring.io/projects/spring-cloud-sleuth
Apache License 2.0
1.77k stars 782 forks source link

java.util.ConcurrentModificationException: null #1511

Closed yechao4j closed 4 years ago

yechao4j commented 4 years ago

java.util.ConcurrentModificationException When i dynamicly refresh RouteDefinition after updating spring.cloud.gateway.routes properties, I found errors below. Maybe i think it's a bug when refresh route under high load(about 10000 QPS). Please anyone help me? My version: SpringCloud version: Greenwich.SR2, Springboot version: 2.1.7.RELEASE

2019-12-23 21:38:33.652 [reactor-http-epoll-23] ERROR o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler - [29f71d5b] 500 Server Error for HTTP GET "/v3/configurations/gray/info?grayTypes=30"
java.util.ConcurrentModificationException: null
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:230)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:933)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.Flux.subscribe(Flux.java:7921)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.Flux.subscribe(Flux.java:7921)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxMaterialize.subscribe(FluxMaterialize.java:40)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxDematerialize.subscribe(FluxDematerialize.java:39)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:229)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515)
        at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247)
        at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2071)
        at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318)
        at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:180)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:70)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64)
        at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226)
        at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:442)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:161)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:745)
yechao4j commented 4 years ago

this is my code:

  public void onChange(ConfigChangeEvent changeEvent) {
    log.info("Refreshing Gateway RouteDefinition!...");
    applicationContext.publishEvent(new EnvironmentChangeEvent(changeEvent.changedKeys()));
    applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this));
    log.info("Gateway RouteDefinition refreshed!");
  }
marcingrzejszczak commented 4 years ago

I don't think it's actually related to Sleuth. Can you confirm that if you disable Sleuth the problem is still there?

spring-projects-issues commented 4 years ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

yechao4j commented 4 years ago

Ok, i will disable Sleuth and try it again. But it's not happened everytime, in most situations it works ok.

yechao4j commented 4 years ago

if i set spring.sleuth.enabled=false, then 

    public void request(long n) {
        try (CurrentTraceContext.Scope scope = this.currentTraceContext
                .maybeScope(this.traceContext)) {
            this.s.request(n);
        }
    }

request method never be invoked, so the problem is not exists. Can this prove the problem is related to Sleuth?

marcingrzejszczak commented 4 years ago

I guess I'd need help from @bsideup or @smaldini with this. I don't think I'm doing anything special about Reactor for that matter?

marcingrzejszczak commented 4 years ago

Would there be an option for you @yechao4j to provide a sample that replicates this issue?

yechao4j commented 4 years ago

I can't reproduce this problem, only i can provide is after 'Refreshing Gateway RouteDefinition!...' log is printed, then 'java.util.ConcurrentModificationException: null' error stacktrace is in endless loop. My springcloud gateway APP runs under high load pressure.

public void onChange(ConfigChangeEvent changeEvent) {
    log.info("Refreshing Gateway RouteDefinition!...");
    applicationContext.publishEvent(new EnvironmentChangeEvent(changeEvent.changedKeys()));
    applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this));
    log.info("Gateway RouteDefinition refreshed!");
  }
marcingrzejszczak commented 4 years ago

If you can't reproduce the problem then how can I help you?

yechao4j commented 4 years ago

emm......

marcingrzejszczak commented 4 years ago

Let me close this issue until you actually figure out how to replicate the error.

zhouyongxiong commented 2 years ago

java.util.ConcurrentModificationException When i dynamicly refresh RouteDefinition after updating spring.cloud.gateway.routes properties, I found errors below. Maybe i think it's a bug when refresh route under high load(about 10000 QPS). Please anyone help me? My version: SpringCloud version: Greenwich.SR2, Springboot version: 2.1.7.RELEASE

2019-12-23 21:38:33.652 [reactor-http-epoll-23] ERROR o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler - [29f71d5b] 500 Server Error for HTTP GET "/v3/configurations/gray/info?grayTypes=30"
java.util.ConcurrentModificationException: null
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:230)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:933)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.Flux.subscribe(Flux.java:7921)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.Flux.subscribe(Flux.java:7921)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxMaterialize.subscribe(FluxMaterialize.java:40)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxDematerialize.subscribe(FluxDematerialize.java:39)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243)
        at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79)
        at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:229)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63)
        at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70)
        at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121)
        at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46)
        at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515)
        at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247)
        at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2071)
        at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318)
        at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:180)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3852)
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:70)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
        at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55)
        at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64)
        at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226)
        at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:442)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:161)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:745)

Hi, I also encountered the same problem. How did you solve it later?

marcingrzejszczak commented 2 years ago

Are you seeing the same problem when using the latest versions?

zhouyongxiong commented 2 years ago

Are you seeing the same problem when using the latest versions?

The currently used gateway version is 2.1.5 Release, reactor core version is 3.2.15 Release, we haven't upgraded to the latest version yet. Shall we upgrade the version of gateway or reactor core?

marcingrzejszczak commented 2 years ago

Try updating Cloud and Boot BOM version to the latest ones

zhouyongxiong commented 2 years ago

Can this problem be solved after upgrading?

marcingrzejszczak commented 2 years ago

I have no idea cause no-one has been complaining and this issue has been opened in 12.2019.