Support for multiple possible status assertions

[smokedlinq]

Problem to solve

Support asserts for status that could return one or more possible status codes that are valid, e.g. 200, 201, 202, 204, 409. The range status check doesn't work when you want to use a complex set like 200, 424, and 503.

Proposal

Option 1 - Add a toString filter so that a matches predicate can be used: status toString matches "^(200|424|503)$"

Option 2 - Add support for a predicate that finds one or more values: status <some_predicate_expression> [200,424,503]; some_predicate_expression could be isOneOf, includedIn, isAnyOf, etc.

Option 3 - Add support for an array of values to write the includes like: [200, 424, 503] has status

Additional context and resources

We have an API endpoint we want to test that could return a different response depending on the state of the system. These are valid responses but we don't want to fail the test as long as it is one of the expected statuses.

[jcamiel]

Interesting problem! We can see how to adapt the current syntax to deal with it.

[fabricereix]

Hi @smokedlinq I like option 2, we could have predicates that take several values

status isOneOf 200 204

[jcamiel]

Another possibility 😊: use or with predicates values. It can works with existing predicates:

GET https://foo.com
HTTP 200
[Asserts]
jsonpath "$.name" == "toto" or "tata"
headers "Content-type" contains "text" or "json" or "csv"
status == 204 or 205 or 206

Remind me also of #347 where we discussed about each, any, none. Would be cool if the two syntaxes play nicely together:

jsonpath "$.values" any == 3
jsonpath "$.values" any > 0 
jsonpath "$.values" any matches "value.*"
jsonpath "$.values" none == 3
jsonpath "$.values" none > 0 
jsonpath "$.values" none matches "value.*"

[fabricereix]

Another possibility: the or could even be applied to the predicate itself This is less readable but more flexible/powerful

status == 200 or == 300   
status < 300 or == 500

[smokedlinq]

How about an in predicate?

status in 200 424 503
jsonpath "$.status" in "Healthy" "Degraded" "Unhealthy"

[jcamiel]

in is cool, the only thing that make me tickle is that all our current predicates are at the 3rd person startsWith, endsWith, contains, includes, matches, exists, isBoolean, isCollection, isDate, isEmpty,isFloat, isInteger, isString... I can also convince me that in is an operator like >, < etc....

[fabricereix]

in seems indeed familiar. I would expect a collection of values after it (in fact it is the same for isOneOf)

For example, reducing the example to one value

jsonpath "$.status" in "Healthy" 

it can be ambiguous. Do we have a collection of one value or the string itself is a collection (like in Python)?

[jcamiel]

Go for in!

[lepapareil]

I agree that in is familiar, but it seems to me that isOneOf is more explicit and therefore more readable by all types of users (non-developers), and above all it's a verb like all Hurl's predicates.

[fabricereix]

Devops that use Python should be happy with in

[lepapareil]

This brings us back to the issue of naming the jsonSchema validation predicate: do we only use verbs or not ? 🤔

[smacktoid]

This feature would be really nice. The scenario I hit recently was where I had some requests which will set up data for later tests.

It would be nice if these setup requests would tolerate the data being in place already, in which case my API will return a 409 status.

So I’d have set the assertion to accept 201 or 409.