Variables in "equals" in expect plugin don't work

[ygilbaum]

I'm trying to run test against local http server that returns json:

{"user":"banana","user_id":15}

While captured variable (uid) is recognized after - get step finished, it is not recognized inside the expect statement. Also, the variable defined in config section is also not recognized in expect/equal statement (is not shown in example). Please advise, what i'm doing wrong, ?

node -v
v11.4.0
artillery -V
1.6.0-26

test.yml:

config:
  target: "http://localhost"
  phases:
    - duration: 1
      arrivalRate: 1
  plugins:
    expect: {}
  variables:
    usr: "banana"

scenarios:
  - flow:
    - get:
        url: "/index.html"
        capture:
          - json: $.user
            as: uid
        expect:
          - statusCode: 200
          - equals:
            - "{{ uid }}"
            - "banana"
        match:
          json: "$.user"
          value: "banana"

    - log: "Usr: {{ usr }}, Uid: {{ uid }}"

altillery returns:

DEBUG=http,http:response  artillery run --quiet test.yml
  http request: {
  "url": "http://localhost/index.html",
  "method": "GET",
  "headers": {
    "user-agent": "Artillery (https://artillery.io)"
  }
} +0ms
  http:response {
  http:response   "date": "Thu, 13 Dec 2018 11:25:32 GMT",
  http:response   "server": "Apache/2.4.18 (Ubuntu) mod_python/3.3.1 Python/2.7.12 OpenSSL/1.0.2g",
  http:response   "last-modified": "Thu, 13 Dec 2018 10:12:12 GMT",
  http:response   "etag": "\"1f-57ce48c51aa14\"",
  http:response   "accept-ranges": "bytes",
  http:response   "content-length": "31",
  http:response   "keep-alive": "timeout=300, max=100",
  http:response   "connection": "Keep-Alive",
  http:response   "content-type": "text/html"
  http:response } +6ms
  http:response "{\"user\":\"banana\",\"user_id\":15}\n" +1ms
  http captures and matches: +5ms
  http { expr: { success: true, expected: 'banana', expression: '$.user' } } +1ms
  http { uid: 'banana' } +2ms
* GET /index.html
  ok statusCode 200
  not ok equals , banana
  expected: all values to be equal
       got: , banana
  Request params:
    http://localhost/index.html
    ""
  Headers:
   date : Thu, 13 Dec 2018 11:25:32 GMT
   server : Apache/2.4.18 (Ubuntu) mod_python/3.3.1 Python/2.7.12 OpenSSL/1.0.2g
   last-modified : Thu, 13 Dec 2018 10:12:12 GMT
   etag : "1f-57ce48c51aa14"
   accept-ranges : bytes
   content-length : 31
   keep-alive : timeout=300, max=100
   connection : Keep-Alive
   content-type : text/html
  Body:
    "{\"user\":\"banana\",\"user_id\":15}\n"
  User variables:
     target : http://localhost
     $environment : undefined
     usr : banana01
     $uuid : 656862b9-3f9c-494f-bd29-75a59584886e
     uid : banana
  http Error: Failed expectations for request http://localhost/index.html
  http     at expectationsPluginCheckExpectations (/usr/lib/node_modules/artillery-plugin-expect/index.js:133:17)
  http     at iteratee (/usr/lib/node_modules/artillery/core/lib/engine_http.js:407:19)
  http     at /usr/lib/node_modules/artillery/node_modules/async/lib/async.js:181:20
  http     at iterate (/usr/lib/node_modules/artillery/node_modules/async/lib/async.js:262:13)
  http     at Object.async.forEachOfSeries.async.eachOfSeries (/usr/lib/node_modules/artillery/node_modules/async/lib/async.js:281:9)
  http     at Object.async.forEachSeries.async.eachSeries (/usr/lib/node_modules/artillery/node_modules/async/lib/async.js:214:22)
  http     at captured (/usr/lib/node_modules/artillery/core/lib/engine_http.js:403:21)
  http     at /usr/lib/node_modules/artillery/core/lib/engine_util.js:401:16
  http     at /usr/lib/node_modules/artillery/node_modules/async/lib/async.js:52:16
  http     at /usr/lib/node_modules/artillery/node_modules/async/lib/async.js:269:32 +8ms

with expect statement commented out:

DEBUG=http  artillery run --quiet test.yml
  http request: {
  "url": "http://localhost/index.html",
  "method": "GET",
  "headers": {
    "user-agent": "Artillery (https://artillery.io)"
  }
} +0ms
  http captures and matches: +9ms
  http { expr: { success: true, expected: 'banana', expression: '$.user' } } +1ms
  http { uid: 'banana' } +2ms
* GET /index.html
User: banana01, Uid: banana

[hassy]

Can you try with the latest version of the plugin? (if you're not using that already) Capturing a variable and comparing it with equals is one of the tests in the test suite, so I'd expect it to work - https://github.com/artilleryio/artillery-plugin-expect/blob/master/test/pets-test.yaml#L28-L38

[ygilbaum]

I think, it's latest. I just installed it today. I'll try to downgrade nodejs version to 10

 npm -g ls --depth=0
/usr/lib
├── artillery@1.6.0-26
├── artillery-plugin-expect@1.2.0
└── npm@6.4.1

[ygilbaum]

@hassy:

in your test you don't use dashes:

      - get:
          url: "/pets/1"
          capture:
            - json: "$.result.0.name"
              as: "name"
          expect:
            statusCode: 200
            contentType: json
            equals:
              - "{{ name }}"
              - "Luna"

the same test with dashes fails in the same way as mine:

      - get:
          url: "/pets/1"
          capture:
            - json: "$.result.0.name"
              as: "name"
          expect:
            - statusCode: 200
            - contentType: json
            - equals:
              - "{{ name }}"
              - "Luna"

I think, the list of expects without dashes is processed only for the first expect, ignoring others.

[hassy]

@ygilbaum yes, you're right - the contents of the expect property can only be an object at the moment (using dashes makes it into a list). I'll mark this as a bug, but in the meantime changing your script to remove the dashes should make it work.

[ygilbaum]

@hassy, It still fails, if I use "equals" as the only expect statement without dashes. If I use it not as the first statement in expect, it's just ignored. Thank you for answering my question.

[hassy]

@ygilbaum just published an update to the plugin which should fix both issues. Thanks for a great detailed bug report!

[ygilbaum]

@hassy, my pleasure. Thank you for excellent product and prompt response! I'll check it shortly.