Added a "succeeded this far" type message to confirm the pipeline logic is running successfully to the implemented stage.
Tightened up the regex (will explain below)
fixed a message generator that was missing an argument
added a suitable namespace to identify the transform after a chat with JJ.
RE regex - the r just means "raw string" so allows you to not care about escaping special characters (like '.'). ^ means start of string, $ means end of string. Without this it can annoyingly match unexpected system files generated during the decompression (i.e it was double matching and finding a file called _pipeline-config.json which we don't care about).
from dpypelines import s3_tar_received
s3_tar_received.start("dp-bleed-ingest-submission-bucket/valid.tar")
you'll probably need to be signed into aws via SSO to do this. If it works you should get an "it worled" type message via a notification to slack (if it doesn't you should get error messages via slack too)
What
Another consolidation pr to tidy things us.
RE regex - the
rjust means "raw string" so allows you to not care about escaping special characters (like '.').^means start of string,$means end of string. Without this it can annoyingly match unexpected system files generated during the decompression (i.e it was double matching and finding a file called_pipeline-config.jsonwhich we don't care about).How to review
Set the 4 env vars from here: https://github.com/ONSdigital/dp-data-pipelines/blob/sandbox/dpypelines/pipeline/shared/notification.py to the webhook (see slack channel or ask in slack for a webhook). literally
export DE_SLACK_WEBHOOK=<the webhook urletc etc in your terminal.run this in a script in the root of the repo:
you'll probably need to be signed into aws via SSO to do this. If it works you should get an "it worled" type message via a notification to slack (if it doesn't you should get error messages via slack too)
Who can review
Anyone with aws SSO.