data_ingress_v1 updated for new configuration

[mikeAdamss]

DEPENDS ON: https://github.com/ONSdigital/dp-data-pipelines/issues/88

What is this

Once we have an updated appraoch to configuration we need to update https://github.com/ONSdigital/dp-data-pipelines/blob/sandbox/dpypelines/pipeline/dataset_ingress_v1.py to use it.

What to do

new function signature

def dataset_ingresss_v1(path_to_files, config):

• The signature needs to expect two things
  • a config entry from the DEPENDS ON task
  • the path to the local files (as it currently does)
• From that starting point rewrite the function, it should mainly be simplification (as we dont need to check for config anymore)
• You'll also need to change the logic for getting required files and supplemntary distributions, there's a "config_version" key in the DEPENDS_ON task that should serve the same purpose and allow us to change the structure of config dicts in the future should we want to.

Acceptance Criteria

• [ ] The function will run using the new config approach.

[mikeAdamss]

done