--no-reset-stats should be on by default

[hhowe29]

Please make this on by default. It is what most people will want in a realistic load test.

Case in point. We are gradually hatching about 1000 locusts that all need to authenticate with a login server. We hatch them slowly to avoid overwhelming the login server. After each locust logs in, it immediately starts performing our regular workflow. In our first test runs, we lost the stats that got collected during the hatch because we didn't know --no-reset-stats existed. Watching your stats reset to 0 is extremely baffling. We had to scour the interweb to learn about this setting, and then spent a few hours figuring out how the switch works in a master/slave scenario

Making this on by default begs the question of how someone would turn it back off if they really do want to reset their stats when the hatch completes. Some options:

1. Ditch --no-reset-stats and replace it with a switch that has the opposite polarity : --reset-stats-on-hatch-complete
2. Just let me hook on_hatch_complete and reset the stats on my own if that is the behavior I want. Assumes that I can do this in my locust class or task set.

[aldenpeterson-wf]

I actually think this would be a good idea. I have fielded this question often.

Curious what other people think. @heyman you have thoughts on this?

[bubbaksmith]

The most confusion/frustrating thing is that no where in the current documentation does it mention that all your data will reset after all Locust have been hatched.

[locmai]

Yeah I always run the tests with --no-reset-stats,I think it would be great if it set by default

[AlexB138]

Agreed, it ought to be the default setting. As previously mentioned, the documentation doesn't spell out what is happening, or indicate that the option to not reset exists. We ended up with many many invalid test results because of this settings before realizing what was going on.

[heyman]

Yeah, I'm all for it!

[heyman]

That is, alternative 1. Turn it off by default, and add a --reset-stats flag.

[hhowe29]

@heyman, I will try to knock out a PR for this

[cgoldberg]

resetting stats may not be intuitive behavior, but you get better results.

I'm curious why anyone would you want to mix metrics from the hatching phase into the main results? Just beware that including this data from before the workload has stabilized, the result distribution can be skewed and any calculated stats may not be representative.

However, I'm fine changing the default as long as we keep a way to reset stats and document the switch to do so.

[bubbaksmith]

I personally feel like more data and a clear procedure would lead to "better results". While using locust, I was unaware that the beginning of my data was being truncated and this lead me to question my data and ultimately not trust it.

Just beware that including this data from before the workload has stabilized, the result distribution can be skewed and any calculated stats may not be representative.

This could also be the sole purpose of a test (to find the point of failure). So if I was stress testing over night with a slow ramp of users, by the time I came back to check my results later they could potentially be truncated.

Im not opposed to the current way of truncating ramp data, but at least clearly tell me that you are doing that. Finding this option in the locust issues as opposed to the Documentation or a UI checkbox was frustrating.

[aldenpeterson-wf]

@cgoldberg in my experience people like comparing "requests made" vs "requests serviced" (from the server) and it's confusing if those numbers differ, which they do because the requests are reset.

[hhowe29]

@cgoldberg I agree that some users may want this behavior. For us though, I expect to make our test server scream in agony long before the hatch is complete. When it does, I want to retain all those stats.

I would expect that many people reaching for locust for the first time are doing so in the context of a test server that is having problems. The most sensible default for those newbs, IMO, is to retain all stats until instructed otherwise. As they use locust more, and start to resolve the problems it uncovers in the test server, they may want to reset stats once they reach a steady state. That seems like more of an advanced used case.

[heyman]

I also want the resetting of stats in most of my use cases. However, I really don't mind using the --reset-stats flag for it, and it has confused new users enough times for me to think that changing the default makes sense.

[spearsem]

@cgbystrom A major use case for not resetting stats is a case when you have a fractional hatch rate.

I work on high-latency, compute-heavy services, so this is the norm for me. My eventual production deployment might involve dozens of servers to scale up to merely hundreds or thousands of concurrent users at a throughput of 1-10 requests per second.

These are paltry numbers compared with large web services handling high volumes of traffic. But the services I work on are used very differently.

As a result, we want to know from load testing how to extrapolate to estimate what resources we will need for a production scale of traffic. It would be prohibitively expensive to actually allocate that much resource to just run the load tests.

So instead, we may run tests on a much smaller scale, maybe even a single server that can handle only 10s of requests per second at a low level of concurrency.

To see the saturation point of a service like this, we would gradually step up the users from 1 to N spread out over a long time, and the stats we collect at each increment when a new user is added are extremely important.