search

openml / server-api

Python-based server
https://openml.github.io/server-api/
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Write developer documentation on writing tests #169

Open PGijsbers opened 3 months ago

PGijsbers commented 3 months ago

After some reading and testing, I found that I think I want to stick with the following strategy for now:

  1. Migration testing, which tests the python api responses against the php ones, when set up against the same pre-populated database. This is so we ensure correctness for migration, and have explicit documentation about the differences (in the unit tests, though they should also end up on the migration page)
  2. Integration testing for expected paths. Simple tests that test the full Python stack, including FastAPI client, dependency injection, and database interactions. For these tests the data will be added to the database on a per-test level. This makes it easy and fast to run tests by any developer- no management of test database state needed. From measurements I did find that creating the TestClient for FastAPI has considerable overhead (approx. 60ms per test). For this reason, we limit this to one test per kind of expected response. E.g., for get_flow: one with a returned flow, one without a result.
  3. Test database functions directly on a database which have items inserted on a per-test level. This is still surprisingly responsive, so there is no need to mock anything out.
  4. Test the broader input/output of the different API methods by calling the Python functions directly instead of through the TestClient. This is so we can test more different scenarios, while being fast. It also ensures that if tests fail at the API level (point 1) but not here, we know it is due to changes of FastAPI its configuration. Database calls at this level can be mocked, to better isolate the input/output processing of this function. I have some consideration of not mocking anything out here, to avoid the extra boilerplate.

This is not set in stone. Perhaps we should not even separate tests out this much, and only factoring out the FastAPI layer is enough. If someone with more experience happens to read this, feel free to leave your thoughts :)