Display response body on assertion failures

[FiloSottile]

Problem to solve

Imagine I have a simple test like

POST http://localhost:7380/sigsum/v1/add-tree-head
[ ... ]
HTTP 200
[Asserts]
body contains "cosignature"

When it fails, this API will print return some useful error message as a text/plain response.

However, if I run it either with --test or without, I will only see an HTTP code failed assertion, with no way to see the body, which I definitely need to debug and triage the test failure.

Proposal

Read some limited amount of response body and if printable, display it on test failure.

[jcamiel]

Hi @FiloSottile I agree and we're going to change the actual behavior: when there is a failed assertion, we should display the body response (probably as if you have run the test with --verbose)

[FiloSottile]

Awesome, thank you for the quick turnaround! ✨

[jcamiel]

@FiloSottile I think I've something working which is a good start. We've decided to add a new command line option --error-format which can take, for the moment, two values: short, long. By default, the asserts are short, without body headers, response etc... This is the current behaviour:

With long error message, we dump the body and headers of the HTTP response:

We can start with this output and tweak it latter. The reason why --error-format long is not the default is due to the potential very long HTTP response. For a start, you should opt-in to have detailed error messages..

(pinging also @dermetfan who runs pipeline with --very-verbose for the same reason)

[gdamjan]

The only thing missing from the long format is the request line. It's a bit hard to know which test failed at test.hurl:249:6 if you have tests running in a CI.

Would that be accepted contribution, or it'll need to be another error-format?

[rromaa7]

It would perhaps be easier to implement and very useful for all developers if there was a flexible way to define a name for each ENTRY that would be output instead of or along with the ENTRY_NUMBER in the event of an error. E.g. in the ENTRY: [Label] or [Entry_name] that would be displayed in an error situation to identify the erroneous request e.g.:

   [Label]
        POST-{{url}} ({{consumerId}})

Then you would get the output:

   ...............................................
   error: Assert failure
   --> /home/user1/projects/pr1/test.hurl:647:0:POST-/api/v1/consumer (99f27808-9a11-bb29-a8d1-1103d46a5186)
647 (POST-/api/v1/consumer (99f27808-9a11-bb29-a8d1-1103d46a5186))    
     jsonpath "$.bezeichnung" == "Test2"
     actual:   string <Test2>
     expected: string <Test2 OK>
   ...............................................