update s3_tar_recieved for new config approach

[mikeAdamss]

DEPENDS ON: https://github.com/ONSdigital/dp-data-pipelines/issues/88

What is this

Once we have an updated appraoch to configuration we need to update https://github.com/ONSdigital/dp-data-pipelines/blob/50846d7d1b8c14a4cced4349144d4b83066bdd02/dpypelines/s3_tar_received.py#L14 to use it.

What to do

• Write a function to use the recieved s3 url to get the correct entry from our new config (see DEPENDS ON task).
• Remove all the pipeline config stuff
• Get the "secondary_function" and call it where we're calling dataset_ingress_v1

Note: you'll need to pass along two positonal arguments to the function at the end. <config>, <path_to_inputs>

Acceptance Criteria

• [ ] Pipeline will run as far as calling the "secondary_function" with the path to the decompressed files as the argument.

[SarahJohnsonONS]

can you set the first example "key" to "^.*$" (anything - so it'd set our wanted default of calling that sdmx transform) if you do pick up the s3 tar received task create an empty function for get_dataset_id or similar that just returns a hard coded "not-specified" (or suchlike for now),. function to get the config based on the key - your "not-specified" will match the default only get the config once, just pass the config dictionary on to the next function

[mikeAdamss]

done