Three functions added (in dpypelines/pipeline/shared/pipelineconfig/transform.py) to get transform inputs, transform function and transform kwargs from pipeline config dict. dataset_ingress_v1() updated to use new functions instead of hardcoded values. Associated tests added in tests/pipelines/pipeline/shared/pipelineconfig/test_transform.py. Tided up some logging statements in dataset_ingress_v1().
How to review
Sanity check, make sure dataset_ingress_v1() runs up to L321.
What
Three functions added (in
dpypelines/pipeline/shared/pipelineconfig/transform.py) to get transform inputs, transform function and transform kwargs from pipeline config dict.dataset_ingress_v1()updated to use new functions instead of hardcoded values. Associated tests added intests/pipelines/pipeline/shared/pipelineconfig/test_transform.py. Tided up some logging statements indataset_ingress_v1().How to review
Sanity check, make sure
dataset_ingress_v1()runs up to L321.Who can review
Anyone.