ONSdigital / dp-data-pipelines

Pipeline specific python scripts and tooling for automated website data ingress.
MIT License
1 stars 0 forks source link

Run the python pipeline code against the existing Supply and Use Data and the GDP examples (Local Execution). #42

Closed martynspooner closed 5 months ago

martynspooner commented 6 months ago

What is this?

Run the transform code against our initial target data products / sources on a local environment. Following the infrastructure build a new work item can be raised for running in the cloud, but this is just for local execution.

What to do:

You need to:

Execute our pipelines against the source targets:

  1. Supply and Use (used for prototyping the transform code).
  2. Published GDP data examples - https://officenationalstatistics.sharepoint.com/sites/ESABCCESSA/DA/Forms/AllItems.aspx?id=%2fsites%2fESABCCESSA%2fDA%2fData_Transmissions%2fONS_Publications%2fSDMX+Test+Files&p=true&ct=1706188252328&ct=1709286454178&or=Teams-HL&or=Teams-HL&ga=1&LOF=1 (6 in total).

Acceptance Criteria

mikeAdamss commented 6 months ago

3