fallback sometimes cause `java.lang.IllegalStateException: Only one connection receive subscriber allowed.` exception

[eatgrass]

SpringCloud Hoxton.SR4 SpringBoot 2.3.0 Release Java version 1.8.0_181 OS: MacOS 10.14.6

Using resilience4j circuit breaker with fallback uri to an external service, sometimes got exception(maybe 5~6 times), no problem with controller in gateway itself. for example

spring:
  application:
    name: api-gateway
  cloud:
    gateway:
      routes:
        - id: timeout_fallback
          uri: http://localhost:4001
          predicates:
            - Path=/fallback
        - id: circuitbreaker_route
          uri: http://localhost:4001
          predicates:
            - Path=/timeout
          filters:
            - name: CircuitBreaker
              args:
                name: c1
                fallbackUri: forward:/fallback

here's the error stacktrace

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
    at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:304) [reactor-netty-0.9.7.RELEASE.jar:0.9.7.RELEASE]
    at reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:139) [reactor-netty-0.9.7.RELEASE.jar:0.9.7.RELEASE]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]

2020-05-22 17:23:57.445 ERROR 8420 --- [ctor-http-nio-3] a.w.r.e.AbstractErrorWebExceptionHandler : [cf015a5a-8]  500 Server Error for HTTP GET "/timeout"

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
    at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:304) ~[reactor-netty-0.9.7.RELEASE.jar:0.9.7.RELEASE]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ HTTP GET "/timeout" [ExceptionHandlingWebHandler]
Stack trace:
        at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:304) ~[reactor-netty-0.9.7.RELEASE.jar:0.9.7.RELEASE]
        at reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:139) ~[reactor-netty-0.9.7.RELEASE.jar:0.9.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]

Sample a simple example to reproduce this problem https://github.com/eatgrass/gateway-fallback

[spencergibb]

Duplicates #1675 fixed by 7ee19ffe800aef9b368ec123702385b9f452c845