search

ONSdigital / dp-data-pipelines

Pipeline specific python scripts and tooling for automated website data ingress.
MIT License
1 stars 0 forks source link

Investigation and Implementation of SDMX 2.1 Transformation Script #129

Closed osamede20 closed 2 months ago

osamede20 commented 2 months ago

What

  1. These SDMX 2.1 script was written to follow the same logic as that of SDMX 2.0. This time, Python ElementTree package could not be applied as xml 2.1 file structure does not provide for easy accessibility of the element attributes. However, the SDMX 2.1 codes still deliver on performance, and the processing time is reasonable.

  2. The flatten_dict from utils.py is called to get the unpacked header dictionary

  3. The validation and tests from task DIS-714 are integrated into the code

How to review

The codes could be checked for performance optimisation They could also be checked for scalability.

Who can review

Anybody

osamede20 commented 2 months ago

All suggested changes have been made and there is a remarkable improvement in the performance of the transform code. Thanks.