Add an api coverage feature

snok / drf-openapi-tester

Test utility for validating OpenAPI documentation

https://github.com/snok/django-openapi-tester

BSD 3-Clause "New" or "Revised" License

119 stars 23 forks source link

Add an api coverage feature #256

Closed anis-campos closed 2 years ago

anis-campos commented 2 years ago

I feel like we could greatly benefit from a coverage tool. The idea is to:

have a percentage of "covered" path, meaning that all possibles requests/responses were validated.
have a percentage of "covered" components, meaning that the object was seen at least once in the responses.

The goal would be to give a good idea of how much of the API is tested and therefore make it possible to identify and address weak spots.

Goldziher commented 2 years ago

How would you implement this?

anis-campos commented 2 years ago

I'm not sure. We need some pytest-cov equivalent of https://github.com/viclovsky/swagger-coverage, but now that I'm thinking about it, it might be completely unrelated to the schema validation done here. The only extra information coming from this tester is the validity of the response, whereas coverage is simply a matter of identifying not tested code.

sondrelg commented 2 years ago

Yeah this sounds like a cool idea, but maybe one that would be better implemented as it's own complementary or standalone package. There would be very little overlap with the existing package internals here, unfortunately.

If you want to experiment and open a PR, I'm not necessarily opposed to a feature like this, depending on how it turns out - but I suspect you would need to essentially write the equivalent of a standalone package, inside drf-openapi-tester, which sounds suboptimal.