rampTo takes longer than specified and ends too early

[CherryDT]

Note: I've previously asked this on Spectrum but got no response for 6 weeks, so I'm escalating this to a GitHub issue because it smells like a bug. If I'm just doing something wrong, let me know.

Hi, I have issues using rampTo. From what I understand, a step with duration: A, arrivalRate: B, rampTo: C is supposed to linearly ramp the rate from B to C over the course of A seconds, right?

However, I consistently get a different behavior which makes it hard to run the tests properly. The ramping step takes several times longer than specified (for example 4 minutes when duration was set to 60) and ramps only to a fraction of the defined rate (for example 30%) only to have artillery then state that the step is supposedly "complete".

Here is an example of my issue:

Config:

  config:
    target: "ws://target"
    phases:
      - duration: 20
        arrivalRate: 100
        rampTo: 2000
      - duration: 20
        arrivalRate: 2000
    ws:
      # Set a custom subprotocol:
      subprotocols:
        - "stuff-0.1.1"
  scenarios:
    - engine: "ws"
      flow:
        - send: "hello"

I'd expect this to ramp from 100 req/s to 2000 req/s over the course of 20 seconds and then continue at 2000 req/s for another 20 seconds.

What happens instead, though, is this: It ramps from 100 req/s to 843 req/s (!) over the course of 60 seconds (!) and then jumps to ~2000 req/s for another 20 seconds. I remove the rampTo phase and start with 2000 req/s right away, it works as expected.

Here an excerpt from the log of this test, showing the issue:

Report @ 18:08:50(+0000) 2020-04-14
Elapsed time: 1 minute, 0 seconds <<<<<<<<<< should take only 20 seconds and not 1 minute
  Scenarios launched:  8422
  Scenarios completed: 8422
  Requests completed:  8422
  RPS sent: 843.89 <<<<<<<<<< should go to 2000 and not 843
  Request latency:
    min: 0
    max: 2.8
    median: 0
    p95: 0
    p99: 0
  Codes:
    0: 8422

Started phase 1, duration: 20s @ 18:08:54(+0000) 2020-04-14
Report @ 18:09:00(+0000) 2020-04-14
Elapsed time: 1 minute, 10 seconds
  Scenarios launched:  16049
  Scenarios completed: 16048
  Requests completed:  16048
  RPS sent: 1606.41 <<<<<<<<<< I assume this is just because the phase just started
  Request latency:
    min: 0
    max: 0.1
    median: 0
    p95: 0
    p99: 0
  Codes:
    0: 16048

Report @ 18:09:10(+0000) 2020-04-14
Elapsed time: 1 minute, 20 seconds
  Scenarios launched:  20000
  Scenarios completed: 20000
  Requests completed:  20000
  RPS sent: 2010.05 <<<<<<<<<< Here it works perfectly
  Request latency:
    min: 0
    max: 1.9
    median: 0
    p95: 0
    p99: 0
  Codes:
    0: 20000

What am I doing wrong?

Thank you!

(This is on 1.6.0)

[LiamDobbelaere]

I had issues with Artillery's rampTo being inconsistent as well during repeated runs. After seeing your issue I got curious and looked in the code.

It seems they use some random probabilities to determine if an arrival happens within a given period for rampTo. I read the code rather quickly, but it would explain the inconsistent results.

  let probabilities = crypto.randomBytes(Math.ceil(spec.duration * 1000 / tick * 1.25));
// (...)
      let prob = probabilities[i++] / 256;
      if (prob <= p) {
        ee.emit('arrival', spec);
      }

(Core/lib/phases.js createRamp)

They did -not- document this correctly iirc and it makes repeated test runs possibly incomparable. I had a much better time just manually writing out the phases myself, at least it was predictable.

[imgalli]

Is this still the case? We're evaluating artillery amongst other options and this kind of thing is a dealbreaker for us.