Version 1.7.0 Breaks Boolean Capture

[RohanNagar]

I have this step that uses the artillery expect plugin and passes in version 1.6.2, but fails in 1.7.0.

- post:
        url: '/verify/reset?email=success@simulator.amazonses.com'
        headers:
          password: 5f4dcc3b5aa765d61d8327deb882cf99
        auth:
          user: application
          pass: secret
        capture:
          - json: $.email.address
            as: email
          - json: $.email.verified
            as: verified
        expect:
          - statusCode: 200
          - hasProperty: creationTime
          - hasProperty: lastUpdateTime
          - equals:
              - "{{ email }}"
              - "success@simulator.amazonses.com"
          - equals:
              - "{{ verified }}"
              - false

It's failing with this error:

not ok equals true, false
  expected: all values to be equal
       got: true, false
  Request params:
    http://localhost:8080/verify/reset?email=success@simulator.amazonses.com
    ""
  Headers:
   date : Wed, 12 May 2021 22:33:46 GMT
   content-type : application/json
   content-length : 254
  Body:
    "{\"email\":{\"address\":\"success@simulator.amazonses.com\",\"verified\":false,\"verificationToken\":null},\"password\":\"5f4dcc3b5aa765d61d8327deb882cf99\",\"attributes\":[\"hello\",\"world\"],\"creationTime\":1620858825642,\"uniqueID\":\"ABC123\",\"lastUpdateTime\":1620858826475}"
  User variables:
     target : http://localhost:8080
     $environment : undefined
     testMetrics : false
     $uuid : ccc8885d-e7d1-4038-b533-1b1f83083dcb
     email : success@simulator.amazonses.com
     creationOriginal : 1620858825642
     errorString : Unable to validate user with provided credentials. (User: success@simulator.amazonses.com)
     creationGet : 1620858825642
     verificationToken : c7bebcfc-89cd-4a98-91bb-343587572a8e
     verified : true

It looks like the capture here is not working correctly:

- json: $.email.verified
  as: verified

Because the JSON body has verified set to false, but artillery thinks it is set to true.

[hassy]

Thank you for the detailed report @RohanNagar!

[RohanNagar]

@hassy any idea where to look for fixing this? Is it within artillery-plugin-expect? I can try to look into fixing this regression but any pointers on where the issue might be would be helpful.

[hassy]

that would be awesome @RohanNagar! the issue is likely in the Artillery itself if the behaviour broke between v1.6.2 and v1.7.2. I'd start by creating a server that returns that exact JSON response you've posted, and running Artillery with DEBUG=* to get debug logs and see where it might be going wrong.

[RohanNagar]

Thanks @hassy. Comparing the debug logs between 1.6.2 and 1.7.2:

1.6.2

http:response {
  http:response   "date": "Thu, 20 May 2021 23:42:54 GMT",
  http:response   "content-type": "application/json",
  http:response   "content-length": "254"
  http:response } +0ms
  http:response "{\"email\":{\"address\":\"success@simulator.amazonses.com\",\"verified\":false,\"verificationToken\":null},\"password\":\"5f4dcc3b5aa765d61d8327deb882cf99\",\"attributes\":[\"hello\",\"world\"],\"creationTime\":1621554173827,\"uniqueID\":\"ABC123\",\"lastUpdateTime\":1621554174094}" +0ms
  engine_util capture: email = success@simulator.amazonses.com +1ms
  engine_util capture: verified = false +0ms
  http captures and matches: +0ms
  http {} +0ms
  http { email: 'success@simulator.amazonses.com', verified: false } +0ms

1.7.2

http:response {
  http:response   "date": "Thu, 20 May 2021 23:45:54 GMT",
  http:response   "content-type": "application/json",
  http:response   "content-length": "254"
  http:response } +0ms
  http:response "{\"email\":{\"address\":\"success@simulator.amazonses.com\",\"verified\":false,\"verificationToken\":null},\"password\":\"5f4dcc3b5aa765d61d8327deb882cf99\",\"attributes\":[\"hello\",\"world\"],\"creationTime\":1621554354178,\"uniqueID\":\"ABC123\",\"lastUpdateTime\":1621554354827}" +0ms
  engine_util capture: email = success@simulator.amazonses.com +1ms
  agentkeepalive sock[0#localhost:8080:](requests: 4, finished: 4) free +78ms
  engine_util capture: verified = false +1ms
  http captures and matches: +0ms
  http {} +0ms
  http {
  http   email: {
  http     value: 'success@simulator.amazonses.com',
  http     strict: undefined,
  http     failed: false
  http   },
  http   verified: { value: false, strict: undefined, failed: true }
  http } +0ms

It looks like the issue is this: failed: true in the line verified: { value: false, strict: undefined, failed: true }. It correctly read false from the JSON but I'm not sure what the failed property indicates?

[hassy]

Ah, I think this line may be the problem:

https://github.com/artilleryio/artillery/blame/v1.6.x/core/lib/engine_util.js#L345 (changed between 1.6 and 1.7)

Because v.value is false (set correctly), the very first check returns true and isCaptureFailed() returns true. Checking that typeof v.value === 'undefined' instead of !v.value should fix it.

If you have a bit of time to test that & package the fix up as a PR that would be awesome. :)

[hassy]

The failed property is the result of isCaptureFailed() on the capture :)

[mkadirtan]

1040 seems related

[RohanNagar]

I'm planning on putting the PR for this together later today.

[RohanNagar]

@hassy I opened #1041 to fix this.