metno / sedr

An experiment to validate OGC EDR APIs using schemathesis, and give good feedback to developers.
GNU General Public License v3.0
0 stars 0 forks source link

Is schemathesis too picky? #2

Closed ways closed 1 week ago

ways commented 1 week ago

Testing a random service, schemathesis will send junk and complain when the server returns 500 or 400 due to the schema only specifying return codes of 200 and 422.

Technically correct is best correct, but this may be creating too much noise when what we want is to help people comply with EDR?

Example:

    | 1. Server error
    | 
    | 2. Undocumented HTTP status code
    | 
    |     Received: 500
    |     Documented: 200, 422
    | 
    | [500] Internal Server Error:
    | 
    |     `Internal Server Error`
    | 
    | Reproduce with: 
    | 
    |     curl -X GET 'https://api.esoh.met.no/collections/observations/position?coords=POINT%2810.72+59.9423%29&parameter-name=air_temperature%3A1.5%3A%2A%3A%2A&datetime=..%2F2023-01-01T00%3A00Z&standard_names=air_temperature&levels=1.0%2F10.0&methods=mean&periods=PT1M%2FPT10M'
ways commented 1 week ago

Discussed with Amund of E-SOH fame. He says "no". =)