Hot thread(reactor-http-epoll) appears when WebClient requests back-end services

[davidyangss]

🐞 Hot thread(reactor-http-epoll) appears when WebClient requests back-end services

• Problem Description I implemented an application using WebFlux. This application provides REST service, and then calls (WebClient) backend service to get the result, and return the result. But, when I stress test, I found a hot thread appeared. Its name is similar to reactor-http-epoll. Through log analysis, I came to the conclusion that almost all Http connections are created by this hot thread, As shown

• Spring version spring-boot-starter-parent 2.5.6 spring-boot-starter-webflux 2.5.6 io.netty 4.1.69.FINAL io.projectreactor:reactor-core:3.4.10 reacotr-pool-0.2.6 reactor-netty:1.0.11

• ReactorResourceFactory

  @Primary @Bean public ReactorResourceFactory reactorClientResourceFactory(PrometheusMeterRegistry prometheusMeterRegistry, CollectorRegistry collectorRegistry) { ReactorResourceFactory resourceFactory = new ReactorResourceFactory(); resourceFactory.setUseGlobalResources(false);

  int maxConnections = Integer.parseInt(System.getProperty(ReactorNetty.POOL_MAX_CONNECTIONS, "500")); // 500 默认
  int pendingAcquireMaxCount = Integer.parseInt(System.getProperty("reactor.netty.pool.acquireMaxCount", "-1")); // -1 默认
  ConnectionProvider provider = ConnectionProvider.builder("RX")
          .maxConnections(maxConnections)  //-Dreactor.netty.pool.maxConnections
          .pendingAcquireMaxCount(pendingAcquireMaxCount) //如果 maxConnections少，pendingAcquireMaxCount要大一点
          //                .maxIdleTime(relayServerProps.getMaxIdleTime())  //-Dreactor.netty.pool.maxIdleTime
          //                .maxLifeTime(relayServerProps.getMaxLifeTime())  //-Dreactor.netty.pool.maxLifeTime
          .metrics(true, () -> new MicrometerPooledConnectionProviderMeterRegistrar(prometheusMeterRegistry))
          //                .pendingAcquireTimeout(relayServerProps.getPendingAcquireTimeout()) //-Dreactor.netty.pool.acquireTimeout
          .build();
  HttpResources httpResources = HttpResources.set(provider);
  resourceFactory.setConnectionProviderSupplier(() -> httpResources);
  resourceFactory.setLoopResourcesSupplier(() -> httpResources);
  
  log.info("ReactorResourceFactory({}) max connections: {}, acquireMaxCount: {}, HttpResources = {} [from ConnectionProvider = {}]",
          resourceFactory, provider.maxConnections(), pendingAcquireMaxCount, httpResources, provider);
  return resourceFactory;

  }

🎁 Reasons for the emergence of hot threads, The hot thread of this stress test is reactor-http-epoll-6

• The root cause of the problem lies in these two classes: reactor.pool.SimpleDequePool and reactor.netty.resources.ColocatedEventLoopGroup
• When reactor-http-epoll-6 runing drainLoop, other threads acquire PoolRef almost at the same time. Therefore, reactor-http-epoll-6 performs a large number of allocators, and allocatorWithScheduler is Schedulers.immediate(). And this allocator is PooledConnectionAllocator.connectChannel return value Publisher
• When TransportConnector.connect, io.netty.channel.Channel is registered to eventloop, and eventloop comes from ColocatedEventLoopGroup. ColocatedEventLoopGroup from DefaultLoopResources.cacheNativeClientLoops. When reactor-http-epoll-6 executes ColocatedEventLoopGroup.next(), it directly fetches the event loop in FastThreadLocal. Therefore, most of the requests are handled by reactor-http-epoll-6.
• Pool acquire and drainLoop

🎁 My advice

• My solution is to no longer use ColocatedEventLoopGroup
• Or, WebClient ExchangeFilterFunction subscribeOn other threads

🎁 The above is my analysis, it may be accurate, please correct me. thanks

[violetagg]

@davidyangss Can you provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.

[davidyangss]

OK, but it will take some time. I will provide a demo

[davidyangss]

I put the demo into the repo: https://github.com/davidyangss/demo.git @violetagg

[davidyangss]

This may be another factor。 When I tested spring-cloud-gateway 3.1.0, I also found the same problem. Especially when the number of connections exceeds 10,000, the problem is obvious

[pderop]

Hi @davidyangss,

Thanks for sharing your sample app.

Are you using "ab" intentionally ? This tool only supports HTTP/1.0 and I got a tons of TIME_WAIT sockets while loading your demo project. Could you use another loader tool which supports HTTP/1.1 + keep alive, like wrk2, gatling, vegeta, or jmeter for example ?

let us know, thanks.

[davidyangss]

Hi, @pderop ,

1. intentionally? no! I use wrk2, the case must appear! I add wrk.sh to the repo "https://github.com/davidyangss/demo.git". "Created a new pooled channel" in one or two threads. According to my understanding, that should be evenly distributed among threads. When I tested spring-cloud-gateway 3.1.0, I also found the same problem.
2. My English is poor, but I still want to describe my analysis of this case. SimpleDequePool is very cleverly designed. But when combined with ColocatedEventLoopGroup, the case will appear. When one thread A executes the drainLoop of SimpleDequePool, many other threads execute doAcquire. Thread A will create resources for other threads. For reactor-netty, thread A is reactor-http-nio. A reactor-http-nio thread will create most of the Http connections. But ColocatedEventLoopGroup use localLoop. So almost all netty events will be concentrated on a reactor-http-nio.
3. The above is my opinion. spring-cloud-gateway 3.1.0 (reactor-netty-core-1.0.13.RELEASE) is also like this. Use reactor-netty-0.8.10.RELEASE, very good.

[pderop]

Hi @davidyangss,

Thanks for having insisted on this issue, I could finally reproduce well the issue using vegeta tool. Now we have a fix available in the 1.0.16-SNAPSHOT, that would be nice if you could give it a try ?