Stats are reset when re-balancing users across slave nodes

locustio / locust

Write scalable load tests in plain Python 🚗💨

https://locust.cloud

MIT License

25.06k stars 3k forks source link

Stats are reset when re-balancing users across slave nodes #1168

Closed heyman closed 4 years ago

heyman commented 4 years ago

When running locust with --reset-stats in distributed mode, the stats are reset every time a new slave connects (when the re-balancing of the simulated users occur). The stats should only be reset when the initial hatching is complete, and not when re-balancing users.

max-rocket-internet commented 4 years ago

I am not using --reset-stats option, I'm using the default docker image locustio/locust:0.13.1 and locust is started like this:

# Slave
/usr/local/bin/python /usr/local/bin/locust --slave --locustfile=/mnt/locust/main.py --host=https://xxxxx.com --master-host=locust-xxx

# Master
/usr/local/bin/python /usr/local/bin/locust --master --locustfile=/mnt/locust/main.py --host=https://xxxxx.com

But I still see this issue 🙂

heyman commented 4 years ago

Hm, that's strange. I'm not able to reproduce the issue without --reset-stats. I guess I have to do some more digging (I'll try and see if I can reproduce it using the Docker image).

max-rocket-internet commented 4 years ago

OK let me know if you want me to debug or test something.

matti commented 4 years ago

I'm also having the same issue as @max-rocket-internet Locust

max-rocket-internet commented 4 years ago

Still seeing this in version 0.13.5:

Screen Shot 2020-01-02 at 16 34 29

carstendev commented 4 years ago

I was able to replicate this or a (related?) issue, by doing the following:

Start a test without --reset-stats with 500 max users and a hatch rate of 1
Lower the number of max users after one minute (still during the hatching phase) to 450

Locust then proceeds to calculate users_to_kill = 500 - 450 and kills 50 users immediately.

Issue here is that as the hatching is still ongoing, and 50 of the at this point only 60 spawned users are killed. I would expect locust to kill users based on the actual spawned users, not just the max setting.

If this is in fact not the same issue, I can also open another issue for this.

heyman commented 4 years ago

I've managed to reproduce the issue using the steps described by @carstendev. It's actually a different issue than the original post, but with similar symptoms.

I'm working on a fix now.

heyman commented 4 years ago

I've now pushed changes (576f7eae2375b454d13499f3b20055d85ae3d301) that should hopefully fix this. Would be great if someone could try out the latest master to confirm that the issue has been fixed.

carstendev commented 4 years ago

I've tried to reproduce the issue with the latest master, but it seems to be fixed now 👍.

max-rocket-internet commented 4 years ago

Awesome, thanks @heyman!