search

tag1consulting / goose

Load testing framework, inspired by Locust
https://tag1.com/goose
Apache License 2.0
792 stars 70 forks source link

assert HTTP status code from goose request? #437

Closed kazimir-malevich closed 2 years ago

kazimir-malevich commented 2 years ago

Are we not supposed to assert HTTP status codes from a Goose request and just view the status_codes in the metrics from the config?

jeremyandrews commented 2 years ago

I'm not sure I'm following your question, but we can't simply return Err() when a Request fails if that's what you're asking, as then on any error a Task wouldn't complete and subsequent requests would be skipped. While sometimes this is desirable, it is not always (for example if you're loading an HTML page and all of its static elements: you don't want the entire Task to exit early if 1 static element 404s).

jeremyandrews commented 2 years ago

You can see validate_and_load_static_assets() for an example of how to work with this: https://github.com/tag1consulting/goose-eggs/blob/main/src/lib.rs#L780

kazimir-malevich commented 2 years ago

Should we be retrieving static elements which would be hosted on a CDN? Wouldn't we just want to validate our system? I was thinking if we made a POST and received 200 that would be seen as a pass.

And also I'd be reticent to have a function that contains 'and' as a function should just do one thing.

We could add an error message on Err() couldn't we and I was thinking of the HTML status code and not the HTML response as a string? But my thinking may be incorrect and confused.

jeremyandrews commented 2 years ago

There are many ways and things to load test. One of the purposes of load tests is to test assumptions: launches can go very badly if in fact the CDN isn't properly configured or if things are flushed too frequently etc. We often leverage Goose to validate our assumptions. That said, Goose is a framework so it's up to you if it just validates the response codes or more.

The linked function is part of a helper library called Goose Eggs, and solves a direct need we have which is to load an HTML page AND then also all contained "static" elements (many of which may actually be dynamically generated). The AND function joins several smaller functions from the Goose Eggs library that can instead be called individually if desired, but this can result in a lot of boilerplate in your load test.

The HTML status code is one of MANY things that Goose can (and does) validate. In some cases you can blindly trust the response code, but it's not uncommon to receive say a 200 OK that is actually a Load Balancer error message, etc. So it's critical that Goose can also validate Headers, Cookies, HTML Titles, and arbitrary text (including JSON) returned by the request. See goose_eggs::ValidateBuilder for a list of things we frequently validate in load tests to be sure things are working as we believe.

kazimir-malevich commented 2 years ago

That was a good answer.