Closed yechao4j closed 4 years ago
this is my code:
public void onChange(ConfigChangeEvent changeEvent) {
log.info("Refreshing Gateway RouteDefinition!...");
applicationContext.publishEvent(new EnvironmentChangeEvent(changeEvent.changedKeys()));
applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this));
log.info("Gateway RouteDefinition refreshed!");
}
I don't think it's actually related to Sleuth. Can you confirm that if you disable Sleuth the problem is still there?
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Ok, i will disable Sleuth and try it again. But it's not happened everytime, in most situations it works ok.
if i set spring.sleuth.enabled=false, then
public void request(long n) {
try (CurrentTraceContext.Scope scope = this.currentTraceContext
.maybeScope(this.traceContext)) {
this.s.request(n);
}
}
request method never be invoked, so the problem is not exists. Can this prove the problem is related to Sleuth?
I guess I'd need help from @bsideup or @smaldini with this. I don't think I'm doing anything special about Reactor for that matter?
Would there be an option for you @yechao4j to provide a sample that replicates this issue?
I can't reproduce this problem, only i can provide is after 'Refreshing Gateway RouteDefinition!...' log is printed, then 'java.util.ConcurrentModificationException: null' error stacktrace is in endless loop. My springcloud gateway APP runs under high load pressure.
public void onChange(ConfigChangeEvent changeEvent) {
log.info("Refreshing Gateway RouteDefinition!...");
applicationContext.publishEvent(new EnvironmentChangeEvent(changeEvent.changedKeys()));
applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this));
log.info("Gateway RouteDefinition refreshed!");
}
If you can't reproduce the problem then how can I help you?
emm......
Let me close this issue until you actually figure out how to replicate the error.
java.util.ConcurrentModificationException When i dynamicly refresh RouteDefinition after updating spring.cloud.gateway.routes properties, I found errors below. Maybe i think it's a bug when refresh route under high load(about 10000 QPS). Please anyone help me? My version: SpringCloud version: Greenwich.SR2, Springboot version: 2.1.7.RELEASE
2019-12-23 21:38:33.652 [reactor-http-epoll-23] ERROR o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler - [29f71d5b] 500 Server Error for HTTP GET "/v3/configurations/gray/info?grayTypes=30" java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:230) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:933) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.Flux.subscribe(Flux.java:7921) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.Flux.subscribe(Flux.java:7921) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:389) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.FluxMaterialize.subscribe(FluxMaterialize.java:40) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.FluxDematerialize.subscribe(FluxDematerialize.java:39) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.Mono.subscribe(Mono.java:3852) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:229) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) at reactor.core.publisher.FluxLiftFuseable.subscribe(FluxLiftFuseable.java:70) at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121) at reactor.core.publisher.FluxLift.subscribe(FluxLift.java:46) at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2071) at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.Mono.subscribe(Mono.java:3852) at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:180) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45) at reactor.core.publisher.Mono.subscribe(Mono.java:3852) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:70) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64) at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226) at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:442) at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:161) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745)
Hi, I also encountered the same problem. How did you solve it later?
Are you seeing the same problem when using the latest versions?
Are you seeing the same problem when using the latest versions?
The currently used gateway version is 2.1.5 Release, reactor core version is 3.2.15 Release, we haven't upgraded to the latest version yet. Shall we upgrade the version of gateway or reactor core?
Try updating Cloud and Boot BOM version to the latest ones
Can this problem be solved after upgrading?
I have no idea cause no-one has been complaining and this issue has been opened in 12.2019.
java.util.ConcurrentModificationException When i dynamicly refresh RouteDefinition after updating spring.cloud.gateway.routes properties, I found errors below. Maybe i think it's a bug when refresh route under high load(about 10000 QPS). Please anyone help me? My version: SpringCloud version: Greenwich.SR2, Springboot version: 2.1.7.RELEASE