Ensure logging works as expected in Databricks job

[JesperGomes]

Synopsis

The settlement report Databricks job does log difference things to application insights. But is it logged under the correct cloud role name? Should we have logging integration tests as we have for CalculatorJob?

Also, we should move logging to a shared package

Acceptance criteria

• [x] Log messages from settlement reports can be found in application insights
• [x] Spans are visible to see in application insights
• [x] Log messages contain the report id
• [ ] Integration test of logging (like for the calculator job) has been implemented for the settlement report job
• [ ] Integration test of logging has been implemented for the shared python package
• [x] Logging functionality is moved to a shared package and is used in both calculator job and settlement report job

Tech Notes

Please see the description on #834. #834 was a duplicate and has been closed.

Tasks

• [x] Add Application Insights connection string for logger
• [x] Add spans to definitions
• [x] Move record_exception to a shared location
• [x] Report id being passed to logger
• [x] Logging functionality is moved to a shared package and is used in both calculator job and settlement report job
• [x] Decorators don't log to Databricks UI. Therefore additional logs are added at the beginning of some methods.
• [ ] Make integration test in settlement reports
• [ ] Make integration test in shared package

[cenh-halfspace]

Screenshot of logging:

Screenshot of spans:

Databricks log: