search

ERDDAP / erddap

ERDDAP is a scientific data server that gives users a simple, consistent way to download subsets of gridded and tabular scientific datasets in common file formats and make graphs and maps. ERDDAP is a Free and Open Source (Apache and Apache-like) Java Servlet from NOAA NMFS SWFSC Environmental Research Division (ERD).
Creative Commons Zero v1.0 Universal
84 stars 58 forks source link

Log all significant events in a standard format #80

Open joefutrelle opened 2 years ago

joefutrelle commented 2 years ago

When ERDDAP is running it doesn't appear to generate a logfile providing information about what operations it's doing and whether or not they're succeeding. I propose that it continuously generate logfiles that

  1. Contain one line for each event, starting with the date and time
  2. Are organized into log levels such as INFO, WARN, ERROR, DEBUG so that they can be configured at an appropriate level for the operational use case

This would enable integrating ERDDAP monitoring and metrics into off-the-shelf systems that perform log analysis, to facilitate real-time monitoring as well as other use cases such as debugging.

BobSimons commented 2 years ago

That's a good idea. It's a big job because there are lots of possible places to generate log messages. I may implement it in parallel with the current logging system because there are also advantages to an unstructured log (e.g., multiline messages).

If you have a suggested log format, please let me know.

joefutrelle commented 2 years ago

I was discussing this with a sysadmin here who pointed me to RFC 5424 which is used for syslogs.

https://docs.ruckuswireless.com/fastiron/08.0.60/fastiron-08060-monitoringguide/GUID-88F338BA-B7BF-485C-B1DE-7418710452A6.html

kfiresmith commented 2 years ago

Hello. I'm the sysadmin who mentioned Syslog enhanced log format (RFC 5424). Thinking about this more, I think that this is likely the job of something like Log4J or any other language-specific logging library. Ideally the logging library would be implemented and configured to create logs in a structure that conforms to the Linux FHS and other norms.

For example, you might create a path at /var/log/erddap, and populate it with something like erddap-master.log in 5424 format, and place other logs alongside it that better handle multi-line (which I believe that syslog can sorta handle so I'd read up on the spec as well...), such as foo-multi.json.

The goal with the main log is to get things into an industry standard format that can be easily forwarded to anyone's log concentration, storage, and analysis system (eg: Splunk, Graylog, ElasticStack, DataDog).

Having things be written in a standard format will allow an on-host forwarder such as rsyslog or syslog-ng to scoop up the lines and ship them off to central log collectors.

That's my quick take at least, as an ops guy.