search

tsenart / vegeta

HTTP load testing tool and library. It's over 9000!
http://godoc.org/github.com/tsenart/vegeta/lib
MIT License
23.51k stars 1.36k forks source link

Different perf-test results between JMeter and Vegeta #503

Closed AndreiPashkin closed 1 year ago

AndreiPashkin commented 4 years ago

Question

We have a web-service that when perf-tested with JMeter tops at X RPS and then just works at X RPS normally with response time within acceptable bounds. But when perf-tested with Vegeta it tops at X/4 RPS and then starts returning 5XX errors and response time goes through the roof.

I wonder - why is that? What is different in how JMeter generates load and Vegeta so that the behavior of our web-service is so different?

tsenart commented 4 years ago

It's possible that JMeter waits for a response before sending the next request, which would mean your service does fine without concurrency. To test this hypothesis, you can do the same with vegeta with the following arguments.

vegeta attack -max-workers=1 -rate=0 ...

This will send requests as fast as possible from a single thread, waiting for a response before sending the next request, like I suspect JMeter behaves.

How does the output of vegeta report look like?

AndreiPashkin commented 4 years ago

@tsenart, I know for sure that JMeter was also launched with about 400 concurrent workers. I don't have the report from Vegeta right now but I'll post it when I get it.

tsenart commented 4 years ago

also launched with about 400 concurrent workers

Yes, but I wonder if it waits for responses before sending the next request. Vegeta always maintains the desired request rate by default, regardless of how slow responses are.

3quanfeng commented 4 years ago

Hi @tsenart , a little bit confusion here, pardon me... if specify -rate=0 which means send new request only after acknowledging the response of last request, doesn't the RPS become even less? or when specify e.g. -rate=1000, the backend cannot handle the concurrency generated by vegeta which causes vegeta become slow to send the requests? Thanks!

AndreiPashkin commented 4 years ago

@3quanfeng

-rate=0 which means send new request only after acknowledging the response of last request

I don't think it means that.

3quanfeng commented 4 years ago

@3quanfeng

-rate=0 which means send new request only after acknowledging the response of last request

I don't think it means that.

but that's what @tsenart said above about -rate=0?

tsenart commented 1 year ago

Without additional evidence, this was most likely due to JMeter slowing down the pace of requests when the server slowed down its responses (aka Coordinated Omission). Vegeta avoids Coordinated Omission. Please re-open if still relevant and there's more data / things to discuss.