search

locustio / locust

Write scalable load tests in plain Python 🚗💨
https://locust.cloud
MIT License
25.06k stars 2.99k forks source link

Random Resets and Invalid Stats #446

Closed bgenchel closed 7 years ago

bgenchel commented 8 years ago

I attempted to run 25 "Users" at a hatch rate of 1 per second. My on_start function has requests to 3 separate routes in it, each of which should be run 1 time during set up, to establish the user entity and it's needed connections.

During run time, the number of requests (as displayed on the webpage/UI) made to each of these routes become desynced, and the requests made to one route severely outnumbers the requests made to others. In addition, the number of requests made to these routes is off in general; by the end of the hatching period, the requests made to each of these routes should match the number of hatched 'Users'. So in my case, should have been 25, but instead i'm seeing 6, 13, 17; Other random numbers smaller than what should be.

On top of that, it seems the UI randomly resets stats. I can see it both on the UI and in the terminal output: test-instance-1/INFO/locust.runners: Resetting stats

It seems that this happens randomly, and it's definitely not what I expected. I'd like to see the pure accurate data displayed to me on the interface.

Is it supposed to reset randomly? When does it reset? If possible, how can I view the accurate data?

sethrosetter commented 8 years ago

I am seeing this as well.

cgoldberg commented 8 years ago

When does it reset?

after all locusts are hatched, stats are reset.

sethrosetter commented 8 years ago

@cgoldberg Ok. I'm pretty new to this obviously. What is the reason for reseting after hatching? Is there a way to disable it?

cedriclecoz commented 8 years ago

reason I am not sure, but to disable it, just comment it out (file runners.py, line 45, self.stats.reset_all())

carn1x commented 8 years ago

Appears to be intended behaviour per https://github.com/locustio/locust/issues/91 , specifically @cgbystrom:

To get to the running state as fast as possible, you need to ramp up your Locusts (users) quickly. This often results in an unnatural behaviour that usually put a lot of stress on your servers. The stress can result in possibly higher response times and error rates. To prevent this, Locust reset the stats to prevent those ramp up stats coloring the running stats.