search

locustio / locust

Write scalable load tests in plain Python 🚗💨
https://locust.cloud
MIT License
25.06k stars 3k forks source link

Performance degradation for constant wait time #1042

Closed jorgebay closed 5 years ago

jorgebay commented 5 years ago

Description of issue

When setting the same value for min_wait and max_wait (greater than 0), the results show response times that are greater than real latency.

Expected behavior

Show real response times, regardless of whether the wait time is constant or random.

Actual behavior

The response times shown are larger than app latency.

Using min_wait = 500 and max_wait = 1500:

p50: ~5ms ✅

Screen Shot 2019-07-03 at 13 03 27

Using min_wait = 1000 and max_wait = 1000:

p50: >20ms ❌

Screen Shot 2019-07-03 at 13 01 33

Environment settings

Steps to reproduce (for bug reports)

Sample locustfile.py:

class StatusBehaviour(TaskSet):
    @task(1)
    def get(self):
        # static page
        self.client.get("/status/")

class WebUser(HttpLocust):
    task_set = StatusBehaviour
    min_wait = 1000
    max_wait = 1000
cyberw commented 5 years ago

Are you 100% sure this is not an issue with the system you are running against? I tried it, but could not reproduce.

max-rocket-internet commented 5 years ago

The response times shown are larger than app latency.

What is app latency here? How are you measuring that?

jorgebay commented 5 years ago

Are you 100% sure this is not an issue with the system

yes, I've tested with ab at the time, the "real latency" is similar to the one shown on the first chart.

cyberw commented 5 years ago

How many users do you run with? Does the problem still occur when running with a single user?

cyberw commented 5 years ago

Also, could you maybe try running ab and locust at the same time?

cyberw commented 5 years ago

Maybe it is possible (but unlikely) that the exactly predictable sleep time makes all the requests happen on the same millisecond (instead of being spread out evenly), which impacts locusts performance, making it have problems with measuring response times correctly.

Are you ramping up very fast?

jorgebay commented 5 years ago

Maybe it is possible (but unlikely) that the exactly predictable sleep time makes all the requests happen on the same millisecond

My guess was along these lines...

How many users do you run with? Does the problem still occur when running with a single user?

I've tested with 200 and 300, using 100 spawns per sec. Maybe that's too fast.

cyberw commented 5 years ago

100/s is high, although I think I've had it that high with no issues. See if you still get the problem with 10/s.

jorgebay commented 5 years ago

It was a cloud deployment that I already teared down...

I can give it a try locally to see if I can consistently reproduce. I was using a EC2 m5.large instance (2 VCPU).