Adds tests and schema validation for report data generation.
JSON schemas for all files were added to tests/schemas. Because the items being tested involve querying BigQuery, it was difficult to do unit tests. I evaluated using tinyquery, however it did not seem worth the effort, as I was able to create a test that verifies the data is correct, and runs a minimal amount of queries in the warehouse.
Additionally, a reports/validate_reports.py file has been created which validates any report data in reports/outputs using the expected JSON schema.
Resolves #205, resolves #107, resolves #147
Type of change
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation
How has this been tested?
Will have github action
[x] create local tests and schema validation
[x] create github action to test the tests and add linting
@atvaccaro great note. Didn't realize pydantic was used so heavily in data-infra. Looks like it wouldn't be too hard to switch to pydantic https://jsontopydantic.com/. Could do that in a follow-up ticket.
Description
Adds tests and schema validation for report data generation.
JSON schemas for all files were added to
tests/schemas
. Because the items being tested involve querying BigQuery, it was difficult to do unit tests. I evaluated using tinyquery, however it did not seem worth the effort, as I was able to create a test that verifies the data is correct, and runs a minimal amount of queries in the warehouse.Additionally, a
reports/validate_reports.py
file has been created which validates any report data inreports/outputs
using the expected JSON schema.Resolves #205, resolves #107, resolves #147
Type of change
How has this been tested?
Will have github action