Propose how to improve testing process

[emmambd]

Describe the problem

We've identified that we want to

• reduce manual testing
• reduce testing pre-release
• clearly identify which test cases are most critical
• Streamline code review assignment

Proposed solution

Develop a clear process for the lifecycle and infrastructure changes we need to make to increase the efficiency of testing and be confident we've covered the critical use cases.

Alternatives you've considered

No response

Additional context

No response

[jcpitre]

Timebox: 1 day

[jcpitre]

To properly cover the test plan we to have data that reproduce conditions that might make the query fail (refer to the Limit problem) Solutions:

1. Improve unit tests to add the conditions
   1. Con: We might have to artificially create a lot of data to reproduce the corner cases
2. Identify data in our current QA or PROD database that reproduce the problem and tailor the query for that
   1. e.g. if we know that the limit problem happens with a certain feed, set the offset so the offending feed is included in the response
   2. Con: The DB changes over time and we might have to readjust the tests if they start failing because of the data
   3. Con: The DB might not contain all the data to reproduce corner cases.
3. Have a special integration test that uses a DB created for that purpose.
   1. The DB would be filled with:
      1. A reduced version of source.csv that we control
      2. The equivalent of test_datasets.json to add missing data that can reproduce error conditions
   2. Con: probably more work

[jcpitre]

Currently we test the API with unit tests and integration tests. Here is a list of tests that should be added. Wether they go in the unit tests or the integration tests is TBD.

• For all feed and dataset endpoints:

  • Test for non-existent ID
  • Verify that the response matches the response schema
  • Test for issue #445, i.e. create the conditions that led to that bug (suspect currently is multiple datasets for a feed, or presence of redirect) ---> _Note that this already has a test added in #460 (See api/tests/test_feedsapi.py)
  • Search by subdivision_name

• /v1/gtfs_feeds

  • Add more combination of parameters
    • We currently do quite a bit of testing. I would suggest adding more combination of parameters (e.g. provider and producer_url, provider and municipalty, etc.)
  • Test for condition that lead to issue #433, i.e. we need a latest dataset that has a null bounding box. ---> Covered in #440
  • Verify that the latest dataset is really the latest
  • Test using dataset_longitudes and dataset_latitudes

• /v1/gtfs_feeds/{id}/datasets

  • Verify that there is only 1 latest dataset
  • Verify that all returned datasets are really for the specified feed

• /v1/datasets/gtfs/{id}

  • Verify the validation report fields (validated_at, features, etc)

• /v1/metadata

  • Verify if the version and commit hash are accurate

• /v1/search

  • Verify the schema of the returned data (should be a combination of gtfs and gtfs_rt feed schemas)
  • Verify that some fields are null depending on the type of result
    • e.g. latest_dataset should be null for gtfs_rt type, entity_types should be null for gtfs type.

[emmambd]

Maybe some "nice-to-haves" that come to mind for me:

• Generate statistics about the total number of feeds that have a latest dataset or more than 1 dataset (would've caught #455)
• Generate stats about the total number of validation reports / total number of datasets (help us catch issues in validator releases)

[emmambd]

Next step is for @jcpitre to create issues for each of the endpoints included here

[emmambd]

Closed and the above issues created to address this.