Develop less-frequent "comprehensive tests"

[maxachis]

Context

• As the number of tests we have expands, it becomes less practical to run all of them (particularly the edge cases) on every PR into dev.
• Many of these tests are error-catching tests which would be captured in various unit tests, but which should ideally be run on every endpoint which uses them, to make sure there are no surprises when they're all tied together.
• Ideally, these would be easy to design and deploy en-masse for various endpoints (for example, it would be trivial to create a test which deploys bad authentication on every endpoint). But they would be time-intensive.

Requirements

• Separate some tests into tests which are run on every pull into main, rather than every PR period.
• These tests would test the less-frequent branches of logic end-to-end (for example, bad parameters, a database failure, authentication failures).

Tests

Test categories include:

• Tests on database integrity not otherwise captured by constraints
• Tests that certain views capture the expected data (for example, that a view which expands on agencies by including some columns from connected tables has the same number of rows as agencies).

Docs

• Will need to develop documentation explaining and differentiating these from other tests.

Open questions

• Any uncertainties? Anything we're flexible about?

[josh-chamberlain]

Nice! Is there a simple way we can delineate in the code which destination branch each test is run on? Seems like most tests would be dev and some would be dev, main.

Would this require new tests, or can we establish this concept with the existing suite and treat further test creation as its own thing?

[maxachis]

Nice! Is there a simple way we can delineate in the code which destination branch each test is run on? Seems like most tests would be dev and some would be dev, main.

Yes! We can either update our existing Github Action or create a separate GitHub action to these more comprehensive tests, which ensures they run on PRs into main.

Would this require new tests, or can we establish this concept with the existing suite and treat further test creation as its own thing?

We could establish this with the existing suite, although most of the tests we have so far do one of the following:

• End-to-end test the happy path of existing endpoints
• Unit test existing functionality (which take very little time)
• Test Database Client methods

All of these I would argue are things we should be testing on every PR, and combined don't take more than 2 minutes.

The new tests I'm envisioning would do a lot more testing, but with the caveat that they'll be testing the same functionality across all the different endpoints, and hence would involve less logic -- it's just checking to see that all the endpoints have the same, consistent behavior.

An example for this is my test_endpoints.py file, which tests that only the HTTP methods we intend for each endpoint return a response, and that others return a "Method Not Allowed" error. It's looking for the same response each time -- it's just testing for consistency.