search

tag1consulting / goose

Load testing framework, inspired by Locust
https://tag1.com/goose
Apache License 2.0
791 stars 70 forks source link

Should the default be one full iteration of the test run when **run-time** is not set without an interrupt signal needed? #467

Closed kazimir-malevich closed 2 years ago

kazimir-malevich commented 2 years ago

If run-time is not set, the test runs for one full iteration but the user needs to ctrl-c to "gracefully stop". 

16:50:30 [INFO] entering GooseAttack phase: Maintain
^C16:51:41 [WARN] caught ctrl-c, stopping...

It's even in the documentation. If a run-time is not set should a single iteration of the the test be run and shutdown of it's own accord?

jeremyandrews commented 2 years ago

If run-time is not set, the intention is to generate load until manually canceled (with ctrl-c). With a load test, it's generally desirable to do things repeatedly so you can get averages, means, deviations, etc ... I've never desired to do a single iteration of the test, but we could certainly explore adding an option to support this. (Maybe: --iterations INT where you can specify exactly how many times to run through entire plan before exiting?)

kazimir-malevich commented 2 years ago

I sincerely note your intentions. But when I am developing a load test, I'd like to know it fully runs before running it for "real". It's unlikely that I'd bring something down if we are hatching by cores, but I don't think interrupt signalling is very clean. A compromise would be --iterations INT, which may add unwitting use. There's plenty of times where a stakeholder in a team would ask me to run the test X number of times, though offhand I am not sure why. But I'm sure there's a use-case.

jeremyandrews commented 2 years ago

I agree with your reasoning: when writing/modifying a load test it's helpful to just run through all Scenarios and Transactions one time to be sure everything is working as expected: I do this fairly manually now and would also find a feature for this useful. While we could do something like --one-shot to run just one time, I think --iterations INT is more flexible.

There'd still be some manual thought required: 1) you'd need to leave the default RoundRobin scheduler enabled, and 2) you'd need to set --users INT to the number of scenarios in your load test. IMO this is fine, and we can simplify/automate this exact configuration in the UI (not yet written).

jeremyandrews commented 2 years ago

If you can test https://github.com/tag1consulting/goose/pull/477 does this address your needs?