Cannot maintain effective debug-level on distributed (parent) logger with local scheduler/cluster

[javabrett]

With config.yaml including simple-form logging config:

logging:
  distributed: debug

>>> import logging
>>> import distributed
>>> logging.getLogger("distributed")
<Logger distributed (DEBUG)>
>>> logging.getLogger("distributed").handlers
[<StreamHandler <stderr> (NOTSET)>]
>>> logging.getLogger("distributed").debug("debug")
distributed - DEBUG - debug
>>> 

All looks good. Start a Client:

>>> from distributed import Client
>>> c = Client()
>>> logging.getLogger("distributed")
<Logger distributed (DEBUG)>
>>> logging.getLogger("distributed").handlers
[<StreamHandler <stderr> (WARNING)>]
>>> logging.getLogger("distributed").debug("debug")
>>> 

No logging output, as the distributed logger has had its Handler (a StreamHandler) set to WARNING threshold.

Once a Client is started and fires-up a local cluster/scheduler, I suspect a worker:

https://github.com/dask/distributed/blob/09b959a5667a51a2dc073510c784e01d44827457/distributed/worker.py#L424-L425

... runs silence_logging, which does just as it is named:

https://github.com/dask/distributed/blob/09b959a5667a51a2dc073510c784e01d44827457/distributed/utils.py#L728-L742

See also #2659 . It might be argued that nothing important is logging to the distributed logger, and everything is logging to a child e.g. distributed.client (even through the loggers are flattened with the simple config), which might be true, but it further erodes the notion of hierarchical logging.

[mrocklin]

@jcrist might have thoughts here

[GenevieveBuckley]

Cross referencing this broader discussion on logging: https://github.com/dask/distributed/issues/2659