Allow users to parameterize test

[ShaddyDC]

Problem to solve

Oftentimes I want to test the same API endpoint with a couple different options and want to verify that they all work correctly. In an ideal world, I'd want to verify for each that the individual expected behaviour is correct, but for simplicity I'm often happy just to verify that it works without causing an error.

Proposal

I think it makes most sense to have variables file-wide rather than per entry. The test would then run over the cartesian product of all options. An example might look like this:

# Parameterized test for creating and verifying a resource

# Define the parameterized variables
type: ["book", "movie"]
status: ["available", "unavailable"]

# Test scenario
POST https://api.example.com/resources
{
    "type": "{{type}}",
    "status": "{{status}}"
}

HTTP/1.1 201
[Captures]
resource_id: jsonpath "$.id"

GET https://api.example.com/resources/{{resource_id}}

HTTP/1.1 200
{
    "id": "{{resource_id}}",
    "type": "{{type}}",
    "status": "{{status}}"
}

Extra useful would be if I could use a set of parameter instances to work together, so I can use a separate component in different places while they still all fit together. Example: type: [("archive", "zip"), ("audio", "mp3"), ("document", "pdf")], and then I can request an "archive" from an api and assert that the returned type is a "zip".

It should also be noted that care must be taken with the cartesian product as the number of tests can quickly explode. Maybe each time a random subset is tried?

Additional context and resources

Templates are already a supported feature. We can manually call hurl on a test with all variable combinations. This just unfortunately isn't great if you have a different set of variables per test and quickly bloats the invocation.

Tasks to complete

• [ ] Decide on a design
• [ ] Implement design
• [ ] Document design

[shawnallen85]

Assuming this will include the hidden issue linked above, is there a status for this enhancement?

[jcamiel]

Hi @shawnallen85,

No advancement for this, and not a priority. It seems to me that these parametized workflows can be done with a proper script without any Hurl syntax addition / modification (something we try to avoid)

[msterin]

I love hurl but it feels that "these parametrized workflows can be done with a proper script without any Hurl syntax addition / modification" is applicable to any part of hurl, as well to the whole hurl. The whole idea (or at least the majority of value to us) is to avoid scripting and have simple but powerful declarative workflow .

We also have a very similar use case - (1) we fetch a list of tenants (in hurl) and validate it (2) we need to iterate over these tenants and execute the same set of tests

so having something to support this without reverting to scripts would make a huge difference for us. I think in 80/20 rule something basic like the suggestion above, or something like the one below, will be very useful. Please consider.

GET https://api.example.com/resources/{{resource_id}} [Options] resource_id: foreach {{a_variable_with_list}} expected_type: foreach {{another_variable_with_list}} HTTP/1.1 200 { "id": "{{resource_id}}", "type": "{{type}}", "status": "{{status}}" }

[hemedani]

You can load easily these parameters from .env files

[msterin]

@hemedani - thanks for the reply . Can you elaborate on it - I am afraid I did not get the suggestion. I need to (1) GET a list, Capture it (e.g. get a list of shard in the storage solution) (2) iterate of the above list, calling the same API (or a set of APIs) for each value (i.e. run the same checks on each shard)

How can I do it with .env files, without popping up into a script ?