Cast phase config strings to numbers in order to support ENV vars

[idpbond]

The Issue

When using ENV-based config for phases, for example:

config:
  phases:
    - duration: "{{ $processEnvironment.DURATION }}"
      arrivalRate: "{{ $processEnvironment.ARRIVAL_RATE }}"
      rampTo: "{{ $processEnvironment.RAMP_TO }}" # The troublemaker

The scenario does not run as expected (finishes early). In my anecdotal setup the rampTo bit is what introduces the issue.

Why does this happen?

The process environment variables which become the spec entry values are strings, and the various mathematical operations on these here: https://github.com/artilleryio/artillery/blob/74baeac42a7bfb9ad9378ce7b9a01f4535eb1490/core/lib/phases.js#L85-L92

And here: https://github.com/artilleryio/artillery/blob/74baeac42a7bfb9ad9378ce7b9a01f4535eb1490/core/lib/phases.js#L118

Are likely the culprit.

The Solution

PR introduces a conversion of defined phase config entries (based on a keys whitelist) expected to have numerical values.

[hassy]

👍 PR merged, thank you!

[xyngfei]

if I checked on the raw report data, I still see the env var assign it as the string, can anyone advise on this issue?

"phases": [
  {
    "name": "ramp up",
    "duration": "10",
    "arrivalRate": "2",
    "maxVusers": "20"
  }

[idpbond]

Thanks for reporting @xyngfei

I'm not sure about the raw report data you pasted and its formatting, but noticed that one of keys (maxVusers) is not one which is cast to a number value.

@hassy I have an open PR that adds maxVusers as a cast value here: #1025

I believe, for now, that it's the only other phase config value in need of parsing, per docs

[xyngfei]

@idpbond, basically I am trying to use environment variables to accept the different config when I enter start the test

phases:

• name: "ramp up"
  duration: "{{ $processEnvironment.art_duration }}" arrivalRate: "{{ $processEnvironment.art_arrivalRate }}" rampTo: "{{ $processEnvironment.art_rampto }}"

However, I noticed when I included the rampTo as the environment variable, the test will not end.

so I am not sure why to have such behaviour especially I am checking the via raw report data, all phases value are casting to string instead of an integer.

[idpbond]

@xyngfei

1. Does this happen on artillery@1.7.2 ?
2. If so, can you provide a stripped down way (CLI or yml) to reproduce the issue?

[xyngfei]

@idpbond Yes, I am using artillery@1.7.2, hope you can able to reproduce this error from your local environment

To run export art_arrivalRate=1 art_rampto=20 art_sustain_duration=30 && ./node_modules/artillery/bin/artillery run --output report.json demo.yml

config:
  target: "https://example.com"
  plugins:
    expect: {}
  phases:
    - name: "ramp up"  
      duration: "{{ $processEnvironment.art_sustain_duration }}"
      rampTo: "{{ $processEnvironment.art_rampto }}"
      arrivalRate: "{{ $processEnvironment.art_arrivalRate }}"
scenarios:
  - name: "test demo site"
    flow:
      - post:
          url: "/api"
          capture:
            - json: "$.success"
              as: success
            - json: "$.errors"
              as: errors
          expect:
            - statusCode: 404

Outcome:

[idpbond]

Hi @xyngfei

Sorry for not double-checking, but it looks like the env-casting feature #978 + #1025 was not yet merged into a tagged release.

I'll have to defer to @hassy as I'm not sure how the folks at artillery do master -> x.y.z release merging

However, if you must use this feature and can't wait for a tagged version, you can do the following:

# Install artillery@master from github directly
npm i -D git://github.com/artilleryio/artillery#master

# demo.yml
config:
  target: "https://example.com"
  # --- Commenting out the plugin ---
  # plugins:
  #   expect: {}
  phases:
    - name: "ramp up"
      duration: "{{ $processEnvironment.art_sustain_duration }}"
      rampTo: "{{ $processEnvironment.art_rampto }}"
      arrivalRate: "{{ $processEnvironment.art_arrivalRate }}"
scenarios:
  - name: "test demo site"
    flow:
      - post:
          url: "/api"
          capture:
            - json: "$.success"
              as: success
            - json: "$.errors"
              as: errors
          expect:
            - statusCode: 404

art_arrivalRate=1 art_rampto=20 art_sustain_duration=30 ./node_modules/.bin/artillery run --output report.json demo.yml

Which, for me, produced a successful (in that it finished) test:

************************************************************

All virtual users done. Summary report:

Total time: 31 seconds
http.request_rate: .................................. 12/sec
http.response_rate: ................................. 12/sec
vusers.created_by_name.test demo site: .............. 160
vusers.created.total: ............................... 160
vusers.failed: ...................................... 161
http.requests: ...................................... 160
http.codes.404: ..................................... 161
http.responses: ..................................... 161
http.response_time:
  min: .............................................. 88.2
  max: .............................................. 354.4
  median: ........................................... 133
  p95: .............................................. 206.5
  p99: .............................................. 262.5

[xyngfei]

@idpbond really appreciate ur speedy response and also an alternative workaround to solve this issue.