search

ONSdigital / dp-data-pipelines

Pipeline specific python scripts and tooling for automated website data ingress.
MIT License
1 stars 0 forks source link

Run the python pipeline code against the existing Supply and Use Data and the GDP examples #62

Closed bailed22 closed 5 months ago

martynspooner commented 5 months ago

Just a few comments and thoughts from Mike and myself:

This tickets intention was to see if the additional 6 GDP files could be executed via the existing SDMX transform code that was prototyped on the Supply and Use file(s).

The transform code is still being worked on to accommodate the latest output specification. Jim worked on the following stubbed / holding code: https://github.com/ONSdigital/dp-data-pipelines/blob/0682f4e8ca72fc5309bbd032c56a85acceeedb92/dpypelines/pipeline/shared/transforms/sdmx/v1.py#L40.

Essentially the transform is a component that takes 1-n positional argument (the SDMX data) and returns a path to two output files (the data and metadata) created during the transform execution.

We need to work with J and his ticket to implement the latest code (that caters for the latest output spec.) and then follow the design that the output paths of the two files are available once the transform has been completed.

martynspooner commented 5 months ago

Main code branch will be updated when the latest output specification code is pushed. David has proven that the GDP examples can be executed. Closing ticket without a merge based on the forthcoming output spec. update impending.

bailed22 commented 5 months ago

All comments cool by me and noted.