openml / server-api

Python-based server
https://openml.github.io/server-api/
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Experiment with `hypothesis` for tests #94

Open PGijsbers opened 1 year ago

PGijsbers commented 1 year ago

Look into using hypothesis to expand test coverage, especially useful for tests which only confirm that the old and new endpoint behave the same (minus documented differences).

PGijsbers commented 1 year ago

Hypothesis does not work nicely with function-level fixtures (c.f. health-checks), which makes it hard to integrate cleanly with the current tests that test POST requests. For those tests, we need clean up and rollbacks to avoid unintended interactions with other tests through persistence in the database. It may be possible with some rewrites, should try initiating the app in main.py from a function, which might allow a separate "app" for each test case to be set up.

PGijsbers commented 11 months ago

But we do currently use it for migration testing with API endpoints that have large search spaces (currently the data/list endpoint).