search

StormSurgeLive / asgs

The Automated Solution Generation System (ASGS) provides software infrastructure for automating coastal ocean modelling for real time decision support, and provides a variety of standalone command line tools for pre- and post-processing. Visit us at https://discord.gg/jFbacxrUf9
https://tools.adcirc.live
GNU General Public License v3.0
39 stars 22 forks source link

clean up logging so that syslog is not cluttered with low impact messages #1085

Closed jasonfleming closed 1 year ago

jasonfleming commented 1 year ago

The writing of messages to the ASGS log file has become excessive to the point that the signal-to-noise ratio has become very low. In particular, the log messages produced by copying files to an external server (usually via output/opendap_post2.sh and especially for frequently updated status messages) are burdensome. The low-impact messages are still important, but should be diverted to a different log file.

wwlwpd commented 1 year ago

For these kinds of things we need to be very specific. Can we be specific about what part we wish to improve? Fielding concerns from users about it being "too verbose" or "too chatty" or whatever should be met with some friendly resistence and a challenge for them to suggest some reasonable behavior. So I guess I am saying, I am skeptical this is actionable and would sooner close this with a "will not fix" than waste time trying to mess around with it.

jasonfleming commented 1 year ago

Specifically, the transmission of the JSON status messages that go out every 1 minute are being logged to $SYSLOG (the transmission of the status messages is logged, not the status itself). These log messages are important, but mainly in the case of something going wrong with the transport mechanism, i.e., they are DEBUG level messages for scp issues. I'm pretty sure these log messages are emitted by opendap_post2.sh. Can we improve the signal-to-noise ratio in $SYSLOG by redirecting these particular DEBUG messages to a different log file?

wwlwpd commented 1 year ago

Thank you for clarifying. I'll keep an eye for this as I get things going for the season. When needed, you really need to be able to go to 1 or 2 places to see what breaks in real time for debugging/troubleshooting purposes or verify that what is happening is expected to be happening.

jasonfleming commented 1 year ago

Yes there is a tradeoff here. I have a few ideas on how to address it. I consider it my responsibility to implement a solution for review.

wwlwpd commented 1 year ago

Will be part of #1156

wwlwpd commented 1 year ago

Definitions of Logs and Log Scopes

  1. console messages - what shows up in the terminal that is running asgs_main.sh
  2. syslog messages - detailed, unambiguous output that can be used to troubleshoot asgs_main.sh type things (important subsystems, but not every little thing)
  3. debug messages - verbose subsystem specific that can be used for detailed troubleshooting of narrowly defined functionality (e.g., opendap_post2.log)
wwlwpd commented 1 year ago

1224 will satisfy this issue when it is completed, so I am closing it.