search

apache / shenyu

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance.
https://shenyu.apache.org/
Apache License 2.0
8.46k stars 2.94k forks source link

[BUG] An error occurred using thread isolation mode in Hystrix Plugin #5577

Open tswc1989 opened 5 months ago

tswc1989 commented 5 months ago

Is there an existing issue for this?

Current Behavior

Use Shenyu V2.6.1 to enable Hystrix. When the isolation mode is semaphore, it is normal. When the isolation mode is thread, similar to this case: https://github.com/apache/shenyu/issues/2893,the error is as follows:

2024-05-31 15:27:40 [parallel-8] ERROR org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin - Response took longer than timeout: PT3S
java.util.concurrent.TimeoutException: Response took longer than timeout: PT3S
    at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$1(AbstractHttpClientPlugin.java:86)
    at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
    at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
    at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2024-05-31 15:27:40 [hystrix-user2-4] ERROR org.apache.shenyu.plugin.hystrix.command.HystrixCommandOnThread - hystrix execute have error: 
org.springframework.web.server.ResponseStatusException: 408 REQUEST_TIMEOUT "Request timeout, the maximum number of retry times has been exceeded"; nested exception is org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
    at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$5(AbstractHttpClientPlugin.java:100)
    at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3797)
    at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3887)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
    at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
    at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
    at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:383)
    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
    at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
    at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
    at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
    at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
    at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
    at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
    at reactor.core.publisher.FluxTimeout$TimeoutOtherSubscriber.onError(FluxTimeout.java:341)
    at reactor.core.publisher.Operators.error(Operators.java:198)
    at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
    at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
    at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
    at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
    at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$4(AbstractHttpClientPlugin.java:97)
    at reactor.util.retry.RetryBackoffSpec.lambda$null$4(RetryBackoffSpec.java:560)
    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:376)
    ... 28 common frames omitted

The configuration of semaphore is as follows:

semaphore

The thread configuration is as follows:

thread

Expected Behavior

Thread isolation mode should be normal

Steps To Reproduce

  1. OS:CentOS Linux 7 (Core)，JDK:openjdk version "1.8.0_412"
  2. The thread configuration is as follows thread
  3. Using Postman to initiate request calls
  4. An error occurred

Environment

ShenYu version(s):v2.6.1

Debug logs

No response

Anything else?

No response

zyd1112 commented 1 month ago

Have you solved it?