config/setting to control log verbosity on HPC UI

[ketzacoatl]

In the current HPC UI, the default log includes lots of DEBUG messages. I would like a way to configure the verbosity, or specify the "log level" (DEBUG/INFO/etc).. the specifics do not really matter so long as we can enable/disable the DEBUG message-flood.

[mgsloan]

This one will be tricky to resolve, since the invoker of the HPC library controls what happens to logs / which logs get logged.

I imagine this can be addressed within the library once log collection is under the purview of HPC. Before then, I could add a utility that handled these details. However, setting the log level based on some external setting is not exactly trivial (when do we check redis? Do we open a new connection for the purpose of getting notified of when the setting changes?)

[ketzacoatl]

I have a few thoughts.

Take this first with a grain of salt (I do not know enough about the domain to speak with as much confidence): could the caller specify a log level, assuming INFO if one is not provided? INFO as a default is sane, and allowing the caller to specify should be sufficient, no?

In regards to the broader topic of architecture, settings, and the like.. typically (via files and environment variables) ops has usually "passed in" this type of configuration/settings info, from outside the application. At this point, some of us in ops are using distributed key-value stores to "share this type of information" with all hosts in the "system at large" (or parts thereof). zookeeper, etcd, and more recently consul, have made this a joy. At worst, on one end of the spectrum, we do this and use a tool to watch the KV for changes and write out specific changes to files, and then, typically reload the service to apply those changes. This breaks continuity of service.

On the other end of the spectrum, we have what I would consider "full-on microservices", where apps are directly connecting and making things happen without the middleman.. apps leveraging the KV directly when it is available (or directly to a stats/log provider, skipping the middleman completely). Some apps only work this way (https://github.com/izenecloud/fiberproxy is an example). This is a simple and clear example of the convergence of devs and ops, and the result is applications supporting these KV backends (and other tools) directly - in other words, applications have begun to see consul and others as their way to more gracefully update themselves (and without requiring the downtime). With a shift to a more micro-services-based architecture. I'm not saying we should do that, but I believe it's worth our while to have this conversation about architecture, and where we are choosing to place ourselves along that spectrum.

[mgsloan]

Yes, this gets a lot easier if the log level doesn't change during runtime. Currently this would be an HMST change rather than a work-queue change, but we could add a utility for this in work-queue.

It'd be great to have some sort of systemic "microservice by default" library. Definitely something to consider!

[ketzacoatl]

Yes, this gets a lot easier if the log level doesn't change during runtime.

I'm not understanding: why would it be difficult to filter log messages written out to log based on the runtime config?

[mgsloan]

I mean like switching log filtering levels while the service is running. Something that could work well would be to be able to change switch the filtering level at a coarse granularity, such as per work item.

[ketzacoatl]

ah, yes. that makes sense. Is that reasonable/doable? I believe that to be a good MVP for this ticket.

[mgsloan]

Yup, that's doable! Should be possible without HMST modifications, actually, by using filterLogger