simple initial logging

[mikeAdamss]

What is this

There have been two recent developers in the pipeline design.

• 1.) We're just sending notifications and a link to slack not a full messahe
• 2.) We're not aiming for full structured logging to kibaba (just yet)

Which means in order for the links we send to slack to have value, the pipeline code must have simple logging statments in them.

What to do

Add some standard python logging statments

Glue is set to capture logs of the INFO level and above as standard, so for our purposes here log to INFO.

As a rule:

• for s3_tar_receive and data_ingress_v1
• anyone we do something (i.e call a function)
• log out the arguments to that functions before its called with an info log
• if it fails, log out everything useful you can with an error log

Note: note the extras field to capture arbitrary information as key value pairs.

Verbose is fine, too much infomrmation is vastly preferable to too little as a starting point.

Once you are happy with the level of logging locally, ask mike to deploy your branch and we'll confirm it works as expected when ran on glue.

How to use the python logging library in this context:

import logging

# Note - for now, this will change,
logger = logging.getLogger(__name__)

# usage example
logger.info("Stuff, things, a message", extra={"some": "info", "some-other": "info"})

Acceptance Criteria

• [ ] Pipeline has logging.
• [ ] Logging working as epxected on glue

[mikeAdamss]

done