Data Quality timeseries alternative flow - Githubissues

Energinet-DataHub / ARCHIVED-geh-charges

Apache License 2.0

7 stars 3 forks source link

Data Quality timeseries alternative flow #1898

Closed Mech0z closed 1 year ago

Mech0z commented 1 year ago

Description:

Make a "flow" to handle quarantined data in Databricks

Rule to implement Consistency" - Is timeseries resolution equal to the metering points resolution for the same period

Acceptance Criteria:

[ ] Find out how to what to do if MP resolution and TS resolution do not match
[ ] Sample data hits Quarantined box
[ ] A secondary flow picks up this data
[ ] The data transformed to valid data
[ ] The data is forwarded to the Gold table