cdriehuys
commented
5 years ago An additional note is that the functional tests are useful for ensuring that we don't make breaking changes to the schemas of the requests or responses for the API. The functional tests should NOT use the project's serializers.
We have had a lot of testing practices over the development of this repository. Our current strategies should be documented somewhere for reference by new developers. In case I don't get around to this, I am going to lay out the general concepts of our current approach below.
Current Approach
Our current approach is to split tests into "functional tests" and "unit tests". At a high level, the functional tests verify that common end-to-end cases work and unit tests verify that each individual component works in isolation for all cases. There should be many more unit tests than functional tests.
Functional Tests
Our goal is to have a functional test for each common use case of the API. For example, it is probably important to test the "happy path" of each endpoint, but it is not important to have a functional test for each endpoint where the user is not authenticated.
An important note about the functional tests is that they primarily interact with the application through HTTP calls. They still have knowledge of the database and how to create objects directly, but their primary method of interacting is through sending and receiving HTTP requests/responses.
In the past we have had a problem with creating too many functional tests. Since these tests involve many more components than an individual unit, we quickly saw an explosion of functional tests as we tried to test every single combination of inputs. Because of this, we have tried to find a happy medium of testing only common paths and leave the testing of every combination to unit tests.
Unit Tests
Unit tests test each component of the application in isolation. These tests are responsible for covering each combination of inputs and outputs. By ensuring that each individual component is well tested, we can be more confident that they will succeed when combined together.
Potential Changes
Here are some things I've been thinking about modifying in our current approach.
More Mocking in Unit Tests
In the past a lot of what we called "unit tests" actually hit the database. We would do things like saving objects and generating the full tree of database objects required to satisfy foreign key requirements. In a few of the newer tests we have been mocking out database access with pretty good results.
The primary benefit of mocking is we actually isolate what we are testing which is something we have been sloppy about in "unit tests" that hit the database. A secondary benefit is we see approximately a 10x speedup when we avoid database access.
The main drawback we've seen so far is that mocking out the database often requires us to know a lot of implementation details when testing. It is still unclear if this is necessarily a bad thing for the unit tests.