locustio / locust

Write scalable load tests in plain Python 🚗💨
https://locust.cloud
MIT License
25.06k stars 2.99k forks source link

Locust freezes after a while (python 3.7) #843

Closed ahmetb closed 6 years ago

ahmetb commented 6 years ago

Hi, I've been using locust to generate some load to for a demo application.

Description of issue

For some reason, after sending a few requests, locust decides to stop doing that.

UPDATE: Switching to python3.6.6 from 3.7.0 does not exhibit this issue.

I was keeping locust version and all the same and my HTTP endpoints still work just fine when I call them manually, locust started to freeze.

I run locust with parameters:

locust --host="http://${FRONTEND_ADDR}" --no-web -c 10 -r 10 

and monitor its output via standard output.

Expected behavior

Locust should continue to run forever since -n is not specified.

Actual behavior

Locust freezes and logs (stdout) stop showing any progress.

[2018-07-15 18:17:56,792] loadgenerator-564c4fdd64-w54vb/INFO/locust.main: Starting Locust 0.8.1
[2018-07-15 18:17:56,793] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: Hatching and swarming 10 clients at the rate 10 clients/s...
 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     0(0.00%)                                       0.00

[2018-07-15 18:17:57,803] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: All locusts hatched: WebsiteUser: 10
[2018-07-15 18:17:57,804] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: Resetting stats

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  0   1(100.00%)       0       0       0  |       0    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     1(0.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  0   1(100.00%)       0       0       0  |       0    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     2(0.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  2    1(33.33%)     168      97     239  |      97    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              2   3(150.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          1     0(0.00%)     488     488     488  |     490    0.00
 POST /cart                                                         1     0(0.00%)     167     167     167  |     170    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/1YMWWN1N4O                                            1     0(0.00%)     299     299     299  |     300    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  2    1(33.33%)     168      97     239  |      97    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              5    3(60.00%)                                       0.00

[... some time passes ..]

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              4     0(0.00%)     124      75     224  |      78    0.00
 GET /cart                                                         10     0(0.00%)     209      68     488  |     210    0.40
 POST /cart                                                        10     0(0.00%)     126      48     315  |     110    0.50
 POST /cart/checkout                                                4     0(0.00%)     326     230     383  |     330    0.10
 GET /product/0PUK6V6EV0                                            6     0(0.00%)     136      69     187  |     130    0.30
 GET /product/1YMWWN1N4O                                            8     0(0.00%)     129      33     299  |     110    0.30
 GET /product/2ZYFJ3GM2N                                            6     0(0.00%)     204     118     341  |     160    0.10
 GET /product/66VCHSJNUP                                            5    1(16.67%)     190     112     282  |     220    0.20
 GET /product/6E92ZMYYFZ                                            9     0(0.00%)     173      64     326  |     160    0.10
 GET /product/9SIQT8TOJO                                            1     0(0.00%)      78      78      78  |      78    0.00
 GET /product/L9ECAV7KIM                                            2    1(33.33%)     215     160     270  |     160    0.00
 GET /product/LS4PSXUNUM                                            6     0(0.00%)     143      34     244  |     110    0.30
 GET /product/OLJCESPC7Z                                           14     0(0.00%)     197      38     397  |     170    0.10
 POST /setCurrency                                                 10     1(9.09%)     130      38     239  |     120    0.20
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                             95     3(3.16%)                                       2.60

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              4     0(0.00%)     124      75     224  |      78    0.00
 GET /cart                                                         11     0(0.00%)     197      68     488  |     210    0.40
 POST /cart                                                        11     0(0.00%)     119      48     315  |     110    0.30
 POST /cart/checkout                                                4     0(0.00%)     326     230     383  |     330    0.10
 GET /product/0PUK6V6EV0                                            6     0(0.00%)     136      69     187  |     130    0.20
 GET /product/1YMWWN1N4O                                            8     0(0.00%)     129      33     299  |     110    0.20
 GET /product/2ZYFJ3GM2N                                            6     0(0.00%)     204     118     341  |     160    0.10
 GET /product/66VCHSJNUP                                            5    1(16.67%)     190     112     282  |     220    0.10
 GET /product/6E92ZMYYFZ                                            9     0(0.00%)     173      64     326  |     160    0.20
 GET /product/9SIQT8TOJO                                            2     0(0.00%)     145      78     212  |      78    0.00
 GET /product/L9ECAV7KIM                                            2    1(33.33%)     215     160     270  |     160    0.10
 GET /product/LS4PSXUNUM                                            7     0(0.00%)     152      34     244  |     200    0.30
 GET /product/OLJCESPC7Z                                           14     0(0.00%)     197      38     397  |     170    0.20
 POST /setCurrency                                                 10     1(9.09%)     130      38     239  |     120    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                             99     3(3.03%)                                       2.20

[freezes here, no more new logs arrive]

Environment settings (for bug reports)

Steps to reproduce (for bug reports)

I'm not sure if I can provide any good repro of this. :( My HTTP endpoint works just fine manually, but Locust stops sending requests.

EDIT: uploaded my locustfile.py here: https://gist.github.com/ahmetb/2645deefff78216933779eeb4043a5e2

ahmetb commented 6 years ago

I reduced -n 10 to -n 5, now instead of freezing at ~100 requests, it made it to 343 requests, only to segfault later:

 Total                                                            343     4(1.17%)                                       0.90

./loadgen.sh: line 10:     6 Segmentation fault      (core dumped) locust --host="http://${FRONTEND_ADDR}" --no-web -c "${USERS:-10}" -r 10 -n 99999

Then after it restarted, now it's stuck again on 233 requests. :(

For posterity, my locustfile is here: https://gist.github.com/ahmetb/2645deefff78216933779eeb4043a5e2

ahmetb commented 6 years ago

After a bit of debugging and switching to python:3.6.6 image, this no longer reproduces. So I'm updating title to match that. Not sure why python:3.7.0 was causing this issue.

aldenpeterson-wf commented 6 years ago

Currently, no one has verified Locust vs 3.7.0 and CI isn't run vs it.