TCP sockets opened lower than requested number of connections

[chrisgleissner]

Version and Runtime

Vegeta 12.11.1, built from vegeta Github repo main branch

Ubuntu 22.04

Expected Behaviour

Number of TCP connections as reported by Linux Sar tool corresponds the number of connections requested from vegeta.

Actual Behaviour

Number of connections is considerably lower in many test scenarios.

For example, 5k connections are requested but only about 1000 TCP sockets are opened.

This is not due to resource constraints on the test machine since for some of the test scenarios, see below, I do reach the full amount of requested connections, e.g. in this test scenario: https://github.com/chrisgleissner/loom-webflux/blob/main/results%2Fcon10k_rps15k_del200ms%2Fwebflux.png

It is almost as though vegeta decides not to ramp up to the full amount of requested connections even though it could.

Neither CPU nor RAM are contended.

Steps to Reproduce

I created a benchmark which relies on vegeta. The system under test happens to be a Java 21 microservice.

You can reproduce the issue by following the instructions on https://github.com/chrisgleissner/loom-webflux

There is a benchmark-all.sh file which allows you to reproduce the issue with a single command, after setup of the required dependencies. It is all self contained and should allow for easy reproduction.

The exact vegeta command used by the benchmark is:

  echo "GET $serviceUrl" | vegeta attack -duration="${_durationInSeconds}"s -rate="$requestsPerSecond" -connections="$connections" -max-connections="$connections" | tee bin/results.bin | vegeta report

The value of $connections was 5000 as well as 10000.

The repo readme contains various charts which showcase the issue. Please have a look at the TCP connections.

Please let me know if you want me to try further test scenarios.

Also, I tried the same with wrk2, and it does not show the issue. However, it does not expose detailed latency results like vegeta, so I would much rather continue to use your excellent tool.

Many thanks.

Additional Context

[tsenart]

The -connections flag controls the maximum open idle connections.

It's poorly named, but kept what it is for backwards compatibility.

Vegeta doesn't offer a way to ensure a fixed number of connections are open — it was made to load tests around requests per second, independent of open connections.

[chrisgleissner]

Thanks for your reply. Much appreciated.

Are there any plans to add this feature?

I based my project on vegeta and being able to simulate a certain number of users is an essential part. I am comparing two alternative frameworks with each other and the validity of the results if impacted if the connections which vegeta creates fluctuates.

Could you maybe produce a pointer to the code which controls the number of connections?

Also, would you be willing to consider merging a PR which adds a new Cli option to to force a constant number of connections?

[tsenart]

No plans. It won't be easy, since http.Transport doesn't let you control the minimum number of open connections, only the maximum.

I'd say it's more useful to think about your system's capacity as number of requests per second.

I'd you can't, then Vegeta isn't the right thing for you likely.

[chrisgleissner]

Thank you.