Open avrohomgottlieb opened 6 days ago
I think this covers most of what we want. Since this issue is a means to an end and not a goal itself we should limit the scope.
We can limit this by saying that for this first version we should have a flag set in the settings to determine if this decorator logging is on or off. Also, since we are mostly interested in timing we should have it output something along the lines of [Start Time] - [Function Name] - [End Time] - [Duration]
It might make sense to also capture the module name where this function is executed.
I think anything beyond that should be considered extraneous. The goal is to keep our logging code in source control but outside of the function implementation. Any additional logging should be completed as we normally log since that is sufficiently tied into implementation.
Context
Issue https://github.com/AlexsLemonade/scpca-portal/issues/944 concluded with the successful generation of computed files on Batch. In order to improve the implementation running on Batch, we need to do some debugging to resolve certain discrepancies and performance bottlenecks that came up during processing.
Next Steps
In order to clarify the reason for the discrepancies and the location of the bottlenecks during the processing, we need an improved approach towards logging. Instead of just littering the codebase with log statements, we'd like to add a logging decorator which will be added to each function in the computed file generation workflow, and can be activated in debug mode when desired.