Glue isnt displaying errors correctly due to our use of the generic Exception message. This task is to make exceptions more specfic to (a) fix the problem and (b) because we need to anyway.
What to do
We need to not be raising generic Exceptions errors. Sometimes a standard python error ("ValueError", "FileNotFoundError") will be fine, in some cases you'll need to create a custom exception for the scenario.
This is how you create a custom exception class in python:
class DecompressFromS3Error(Exception):
"""
Raised in the event we were unable to
decompress a source file.
"""
def __init__(self, msg: str):
self.msg = msg
Then import that error and use it instead of the generic exception.
A few points:
We need to do this for all the genericException exceptions we're raising.
Only create a custom exception where its a unique scenario, i.e if the standard ValueError fits the bill then use that.
Where you create a custom error then reuse if you can, creating a custom error that makes sense in more than one place that's great.
mikeAdamss
commented
3 months ago
What is this
Glue isnt displaying errors correctly due to our use of the generic
Exceptionmessage. This task is to make exceptions more specfic to (a) fix the problem and (b) because we need to anyway.What to do
We need to not be raising generic
Exceptionserrors. Sometimes a standard python error ("ValueError", "FileNotFoundError") will be fine, in some cases you'll need to create a custom exception for the scenario.This is how you create a custom exception class in python:
Taking this error as an example: https://github.com/ONSdigital/dp-data-pipelines/blob/810acdec4b543d01c8dc078df13edd23f265504a/dpypelines/s3_tar_received.py#L32
dpypelines/oipelines/errors.pyThen import that error and use it instead of the generic exception.
A few points:
Exceptionexceptions we're raising.ValueErrorfits the bill then use that.Acceptance Criteria