ovis-hpc / ldms

OVIS/LDMS High Performance Computing monitoring, analysis, and visualization project.
https://github.com/ovis-hpc/ovis-wiki/wiki
Other
98 stars 50 forks source link

reordering of subsystem log statements defeats log_level commands for debugging #1342

Open baallan opened 9 months ago

baallan commented 9 months ago

For the input

log_level level=WARNING
load name=${plugname}
log_level name=sampler.dstat level=INFO
config name=${plugname} producer=localhost${i} schema=${plugname} instance=localhost${i}/${plugname} component_id=${i}
start name=${plugname} interval=1000000 offset=0
log_level name=sampler.dstat level=DEBUG

the driver reorders the statements automatically to:

 log_level level=WARNING(null)
 log_level level=INFO name=sampler.dstat
 log_level level=DEBUG name=sampler.dstat
 load name=dstat
 config name=dstat producer=localhost1 schema=dstat instance=localhost1/dstat component_id=1
 start name=dstat interval=1000000 offset=0

as seen when starting ldmsd with an option like: "-L 54:${LOGDIR}/log_config.1"

When loading conf from the command line, we need to promote only the very first log_level statement which does not name a subsystem and must not promote log_level statements with named subsystems.

tom95858 commented 4 days ago

@baallan, this sounds incorrect to me. The log_level statements that specify the subsystem should have precedence over the default.

baallan commented 4 days ago

promoting all log_level statements to before any load statements breaks the plugin-specific ones (with the current implementation, the plugin-specific logs do not exists, which turns the requested plugin log_levels into either no-ops or errors about facilities that do not yet exists. If promoting all log level statements (which i don't advocate) we need the ability to forward declare log levels for not-yet-loaded plugins.