The-Academic-Observatory / academic-observatory-workflows

Telescopes, Workflows and Data Services for the Academic Observatory
https://academic-observatory-workflows.readthedocs.io
Apache License 2.0
16 stars 0 forks source link

INF-665: OpenAlex Telescope - Schema detect feature #198

Closed alexmassen-hane closed 7 months ago

alexmassen-hane commented 8 months ago

As there are constantly issues with OpenAlex changing their data structure, I have added a feature that will scan through each transformed record to generate a Biguqery style schema.

Tasks have been added that will upload the generated schema to the transform folder and compare it against the schema that we've defined for each entity. If there are any differences between the two it will send a Slack notification for the entities that have any differences.

codecov[bot] commented 8 months ago

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (f252a4f) 74.38% compared to head (50a5fa0) 93.97%. Report is 143 commits behind head on main.

Files Patch % Lines
academic_observatory_workflows/clearbit.py 85.71% 1 Missing and 1 partial :warning:
academic_observatory_workflows/config.py 91.30% 1 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #198 +/- ## =========================================== + Coverage 74.38% 93.97% +19.59% =========================================== Files 27 22 -5 Lines 3697 5498 +1801 Branches 493 761 +268 =========================================== + Hits 2750 5167 +2417 + Misses 844 194 -650 - Partials 103 137 +34 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.