The-Academic-Observatory / oaebu-workflows

Telescopes, Workflows and Data Services for the 'Book Analytics Dashboard Project (2022-2025)', building upon the project 'Developing a Pilot Data Trust for Open Access eBook Usage (2020-2022)'
https://documentation.book-analytics.org/
Apache License 2.0
5 stars 0 forks source link

BAD-308: schema and BQ description updates #116

Closed keegansmith21 closed 1 year ago

keegansmith21 commented 1 year ago

The schema files can differ from the BQ tables since they are not used in their creation. This has led to the divergence of the BQ table schema and the documentation schema.

This update uses the PR from The-Academic-Observatory/observatory-platform#589 to make use of the schema.json files when creating the BQ tables. This accomplishes two desirable things:

As a result, the workflows will now issue an error when a SQL query creates a table that does not fit the expected schema. You may still neglect to supply a schema file, which will not raise any issues and upload the data with an implied schema.

codecov[bot] commented 1 year ago

Codecov Report

Base: 94.83% // Head: 94.86% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (bf8f153) compared to base (ddde9b0). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #116 +/- ## =========================================== + Coverage 94.83% 94.86% +0.02% =========================================== Files 23 23 Lines 2635 2650 +15 Branches 340 340 =========================================== + Hits 2499 2514 +15 Misses 70 70 Partials 66 66 ``` | [Impacted Files](https://codecov.io/gh/The-Academic-Observatory/oaebu-workflows/pull/116?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory) | Coverage Δ | | |---|---|---| | [oaebu\_workflows/workflows/oapen\_workflow.py](https://codecov.io/gh/The-Academic-Observatory/oaebu-workflows/pull/116/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory#diff-b2FlYnVfd29ya2Zsb3dzL3dvcmtmbG93cy9vYXBlbl93b3JrZmxvdy5weQ==) | `92.73% <100.00%> (+0.12%)` | :arrow_up: | | [oaebu\_workflows/workflows/onix\_telescope.py](https://codecov.io/gh/The-Academic-Observatory/oaebu-workflows/pull/116/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory#diff-b2FlYnVfd29ya2Zsb3dzL3dvcmtmbG93cy9vbml4X3RlbGVzY29wZS5weQ==) | `91.54% <100.00%> (+0.05%)` | :arrow_up: | | [oaebu\_workflows/workflows/onix\_workflow.py](https://codecov.io/gh/The-Academic-Observatory/oaebu-workflows/pull/116/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory#diff-b2FlYnVfd29ya2Zsb3dzL3dvcmtmbG93cy9vbml4X3dvcmtmbG93LnB5) | `92.86% <100.00%> (+0.12%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

keegansmith21 commented 1 year ago

Reopening this PR because I've changed the names of some of the schema files (removed dates).

This changed is blocked by The-Academic-Observatory/observatory-platform#589

keegansmith21 commented 1 year ago

All tests passing locally