The current code uses the Trace class, defined in lib.py, to generate diagnostic messages and standard output. Trace could be flexible enough to include logic and filters and handlers, but this is reinventing the wheel. There's already the logging python module, and we should migrate trace messages to that.
We need standardization across the different redux modules in how we trace, log, and output messages, and we need flexibility to tune that output without going into the source code, and we need the diagnostic/debug messages to not cause undue overhead when they are disabled in config.
low-level diagnostics are important in debugging code and developing the algorithms, but the amount of output needs to be tunable outside of the code
Part of this issue also means replacing any remaining print() statements in the code and creating a logging.conf file.
Another part of this issue is making use of the module API such that there is no string formatting done when the message is not to be output. Current trace messages format the output string every time, which is problematic if the logging is at a low level (such that formatting overhead is incurred for very many logging calls - perhaps 10's of millions of calls, in the case of something like vcf call info).
logging is a standard python module, so there is plenty of documentation
example (this is just a demo and there are many possible implementation choices TBD):
import logging
class BraceString(str):
def __mod__(self, other):
return self.format(*other)
def __str__(self):
return self
class StyleAdapter(logging.LoggerAdapter):
def __init__(self, logger, extra=None):
super(StyleAdapter, self).__init__(logger, extra)
def process(self, msg, kwargs):
if kwargs.pop('style', "%") == "{": # optional
msg = BraceString(msg)
return msg, kwargs
root = logging.getLogger()
root.setLevel(logging.DEBUG)
bf = logging.Formatter('{asctime} {name} {levelname:8s} {message}', style='{')
handler=logging.StreamHandler()
handler.setFormatter(bf)
root.addHandler(handler)
kits = [0,1,2]
kitID = 'N4826'
logger = StyleAdapter(logging.getLogger('redux.py'))
logger.info("num kits is {0}", len(kits), style="{")
# 2018-03-21 14:16:52,339 redux.py INFO num kits is 3
logger.info("kit name is %s", kitID, style="%")
# 2018-03-21 14:18:16,080 redux.py INFO kit name is N4826
The current code uses the Trace class, defined in lib.py, to generate diagnostic messages and standard output. Trace could be flexible enough to include logic and filters and handlers, but this is reinventing the wheel. There's already the logging python module, and we should migrate trace messages to that.
We need standardization across the different redux modules in how we trace, log, and output messages, and we need flexibility to tune that output without going into the source code, and we need the diagnostic/debug messages to not cause undue overhead when they are disabled in config.
low-level diagnostics are important in debugging code and developing the algorithms, but the amount of output needs to be tunable outside of the code
Part of this issue also means replacing any remaining print() statements in the code and creating a logging.conf file.
Another part of this issue is making use of the module API such that there is no string formatting done when the message is not to be output. Current trace messages format the output string every time, which is problematic if the logging is at a low level (such that formatting overhead is incurred for very many logging calls - perhaps 10's of millions of calls, in the case of something like vcf call info).
logging is a standard python module, so there is plenty of documentation
example (this is just a demo and there are many possible implementation choices TBD):