search

agrostar / zzapi-vscode

VS Code extension for zzapi - an API testing and documentation framework
MIT License
3 stars 0 forks source link

Run status and header tests first so that body tests can be skipped #5

Closed vasan-agrostar closed 4 months ago

vasan-agrostar commented 4 months ago

Forked from issue #3 by @gitblit

I'd like to see status and $h tests run first, always.

Currently if you define those in common they appear to be appended to the endpoint test list and run last. Perhaps prepend the common tests (in the order defined by the user) to the endpoint test list OR perhaps sort status & header tests first, regardless of where they are defined?

If status or $h tests fail, the $. body tests are likely to fail since the response shapes or types will be different than expected. Perhaps do not run $. tests if the status and/or $h. tests fail? I have example endpoints where 400 or 403 responses return strings instead of application/json and this creates noise when zzzapi attempts to parse JSON in these cases.

vasan-agrostar commented 4 months ago

An option to stop further tests on the first failure would be very useful in addition to running common tests first and then the per-request tests.

vasan-agrostar commented 4 months ago

Let us run the tests in the following order:

  1. Status test
  2. Header tests
  3. Body tests

If there are tests defined in common as well as per-request, let us run the common ones first (no particular reason I can think of, but it seems intuitive). The order will only affect the "stop on failure" option.

But then we probably need to run all the body tests if the status and header tests pass. So it's best to have the option affect only status failures for now. I can't think of a case where a header test failure can mean the request totally failed, so it's pointless to run the body tests.

vasan-agrostar commented 4 months ago

zzapi v1.2.0 and zzapi-vscode v2.1.0 address this issue.

gitblit commented 4 months ago

@vasan-agrostar how do I use stopOnFailure? VS Code allows me to specify it in the schema but running all requests or executing with CLI both fail.

vasan-agrostar commented 4 months ago

A bug was introduced, we need to make a new release. I will do it sometime today.

vasan-agrostar commented 4 months ago

zzapi-vscode 1.2.1 should fix the problem.

gitblit commented 4 months ago

Confirmed. This is now not-failing on CLI and VSCode plugin.

I still have an issue where a body test expects & attempts to deserialize application/json for a 201 result but the endpoint is actually responding 403 and text/plain - response is "Forbidden". One motivation for fail-early was to avoid the attempt to JSON-parse a non-JSON response.

5/17/2024, 8:12:32 AM [ERROR] POST Creates Person status: 403 size: 9 B time: 6 ms parse error(Unexpected token F in JSON at position 0)

Maybe there should be some automatic, built-in validation to assert the response content-type is application/json before attempting to JSON deserialize?