Performance degradation with a high request rate and multiple concurrent workers

[GaelGirodon]

Description

During a load testing using vegeta, when I try to send 500 requests/sec with up to 50 concurrent workers, the latency becomes much higher and I get some connection timeouts (504 Gateway Time-out):

Req/s / Workers	1 / 1	10 / 1	50 / 10	100 / 20	500 / 50
Mean latency	2ms	2ms	1ms	1ms	50ms

• With Reactor Netty 0.9.4, the latency is lower but I get PrematureCloseException errors and 500 HTTP responses (see reactor/reactor-netty#1038)
• With Reactor Netty 0.9.4 and a fixed pool, it's getting even worse

Maybe I'm doing something wrong but I've done approximatively the same test with an old Camel/CXF stack running on Apache Karaf and I don't have these performance drops.

Sample

• Create an empty Spring Cloud Gateway project
• Upgrade Reactor Netty to 0.9.6.RELEASE
• Add a route like this one in the main configuration file:

spring:
  cloud:
    gateway:
      routes:
        - id: test-api
          uri: http://localhost:3000 # A super-fast (Go) local stub server
          predicates:
            - Path=/test/api/**
            - Method=GET
          filters:
            - RewritePath=/test/api/?(.*), /api/v1/$\1

• Run the gateway behind a reverse proxy (Nginx in my case)
• Load test the gateway from another VM with a high number of requests/sec and many concurrent workers, e.g.:

echo 'GET http://ip:port/test/api' | ./vegeta attack -duration=10s -rate=500/1s -max-workers=50 | ./vegeta report -type=json

Environment

• Spring Cloud Gateway version : 2.2.1 (with Reactory Netty 0.9.6)
• JVM version: OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
• OS and version: CentOS Linux release 7.7.1908

[GaelGirodon]

Performance degradation seemed to be linked to having a reverse proxy (Nginx) in front of the gateway, on the same machine (see https://github.com/reactor/reactor-netty/issues/1038#issuecomment-603877063). Results are much better when Nginx is bypassed:

Req/s / Workers	1 / 1	10 / 1	50 / 10	100 / 20	500 / 50	1000 / 100
Mean latency (ms)	1.7	1.5	1.3	1	0.9	2.1

Any explanation about this behavior is welcome!

[tony-clarke-amdocs]

Doesn't it suggest that nginx is the bottleneck?

[spencergibb]

Closing, see https://github.com/reactor/reactor-netty/issues/1038 for a similar closing.

[GaelGirodon]

@tony-clarke-amdocs

Doesn't it suggest that nginx is the bottleneck?

Yes and no. It would be a bit pretentious to say that Nginx is having (or is causing) performance issues 😉 It seems that my problem was to have a reverse proxy (Nginx) and the Spring Cloud Gateway app on the same machine. But why is this causing problems? Too many opened connections within the same machine, I/O limitations? I don't know exactly how to explain this behavior.