Orca lead - It seems like each step of the test is still running by too fast

[lancelakey]

If I do 25+ repeats then the whole test does last longer but this doesn't seem to be generating the kind of / speed of test runs I'd like

I'm currently running:

quorum: 50
repeat: 25
step: 1

I HATE the terms repeat and step here. For clarity I'm replacing step with increment and replacing repeat with step.

Perhaps I'd like another value, a repeat value:

quorum: 50
start: 100
step: 25
increment: 1
repeat: 10

In this case repeat would mean "repeat each step/increment run 10 times"

This is kind of complicated when you think about how really all I want is to run a test for 10 minutes that goes from something, 1, 100, whatever, up to a few thousand concurrent requests. I don't think I give a shit about increments and steps and repeats.

Because these Orca runs still seem to go by pretty fast I've been running them inside a for loop:

for i in {1..20} ; do /path/to/bin/lead -c /path/to/load-test.cson ; done

[dyoder]

The word step used this way doesn't make any sense to me. It's a relatively small point, but step is frequently used to mean "increment by." If you'd prefer to use increment that's scans, but not if you're also using step.

As a matter of process, I think it's generally best to consider the possibility that specific terminology was selected for a reason. Sometimes that's not the case, but just assuming that is sort of aggressive. In this case, I had chosen step and repeat "for clarity" already. I'm not arguing that it was clear to everyone, just that there was some thought put into the use of those terms.

For example, to quote the CoffeeScript docs on loops (emphasis added):

To step through a range comprehension in fixed-size chunks, use by ...

The word repeat is similarly commonly used to describe looping constructs, which is what I was trying to evoke here: "Do this test [repeat] times, incrementing concurrency by [step]."

Again, I'm not arguing for this terminology, just against the mindset of "this bothers me so I'm going to fix it" instead of trying to understand the original intent.

In this case, having increment and step doesn't seem to me to add clarity.

As to "not giving a shit about increments and steps and repeats", I think that could have just been a separate ticket to add some defaults. It's ultimately useful to be able to control these variables when trying to pin down the point where performance begins to degrade, but I can see why defaults would be useful when you just want a baseline.