AlexsLemonade / scpca-portal

Single-cell Pediatric Cancer Atlas Portal is a growing database of uniformly processed single-cell data from pediatric cancer tumors and model systems
https://scpca.alexslemonade.org
BSD 3-Clause "New" or "Revised" License
3 stars 0 forks source link

Add logging decorator for use in debugging #961

Open avrohomgottlieb opened 6 days ago

avrohomgottlieb commented 6 days ago

Context

Issue https://github.com/AlexsLemonade/scpca-portal/issues/944 concluded with the successful generation of computed files on Batch. In order to improve the implementation running on Batch, we need to do some debugging to resolve certain discrepancies and performance bottlenecks that came up during processing.

Next Steps

In order to clarify the reason for the discrepancies and the location of the bottlenecks during the processing, we need an improved approach towards logging. Instead of just littering the codebase with log statements, we'd like to add a logging decorator which will be added to each function in the computed file generation workflow, and can be activated in debug mode when desired.

davidsmejia commented 3 days ago

I think this covers most of what we want. Since this issue is a means to an end and not a goal itself we should limit the scope.

We can limit this by saying that for this first version we should have a flag set in the settings to determine if this decorator logging is on or off. Also, since we are mostly interested in timing we should have it output something along the lines of [Start Time] - [Function Name] - [End Time] - [Duration] It might make sense to also capture the module name where this function is executed.

I think anything beyond that should be considered extraneous. The goal is to keep our logging code in source control but outside of the function implementation. Any additional logging should be completed as we normally log since that is sufficiently tied into implementation.