Open mikeAdamss opened 3 months ago
Add logging statments to data_ingress_v1.
These are the docs for the logger: https://github.com/ONSdigital/dp-python-tools/tree/develop/dpytools/logging
And I've added some quick examples of using it here: https://github.com/ONSdigital/dp-data-pipelines/blob/logging-trial/dpypelines/s3_tar_received.py
There's no "bad" place to add logs but its hekpful to have a consistent pattern, so for this task you're looking to log out:
This does change the approach in the try catches, so this:
try: decompress_s3_tar(s3_object_name, "input") except Exception as err: de_notifier.failure() raise Exception( message.unexpected_error( f"Failed to decompress tar file {s3_object_name}", err ) ) from err
becaomes something like this
try: decompress_s3_tar(s3_object_name, "input") logger.info("S3 tar recieved successfully decompressed to ./input", data={"s3_object_name": s3_object_name}) except Exception as err: logger.error("Failed to decompress tar file", err, data={"s3_object_name": s3_object_name}) de_notifier.failure() raise err
the overarching point being we dont need the verbose expression of what went wrong as the logger will capture and format that information for us.
note: no tests needed for this, the logger has tests already.
What is this
Add logging statments to data_ingress_v1.
What to do
These are the docs for the logger: https://github.com/ONSdigital/dp-python-tools/tree/develop/dpytools/logging
And I've added some quick examples of using it here: https://github.com/ONSdigital/dp-data-pipelines/blob/logging-trial/dpypelines/s3_tar_received.py
There's no "bad" place to add logs but its hekpful to have a consistent pattern, so for this task you're looking to log out:
This does change the approach in the try catches, so this:
becaomes something like this
the overarching point being we dont need the verbose expression of what went wrong as the logger will capture and format that information for us.
note: no tests needed for this, the logger has tests already.
Acceptance criteria