search

dbuenzli / logs

Logging infrastructure for OCaml
http://erratique.ch/software/logs
ISC License
86 stars 19 forks source link

Question on best approach to structured logging (json output) #24

Closed Lupus closed 5 years ago

Lupus commented 5 years ago

I wanted to have fancy colourful text logging output when running my app interactively, but on the other hand nothing is better than json logs for production instance, so that they can be fed into Elastic or some other log processing system.

Also I wanted to embed some machine-readable fragments to each log message, in a form of json blob for simplicify, and have that added to the end of log messages when running interactively, and blend into log json object when running in production mode.

An example of what I'm trying to achieve:

1558955386.283293 [INFO] Application instance #10 has launched        {"instance_num": 5}
{"ts": "1558955386.283293", "log_level": "info", "message": "Application instance #10 has launched", "data": {"instance_num": 5}}

Probably I need 2 reporters, one for human readable output, another for json. Structured metadata of reach message should probably be a tag with Yojson.t inside (does that sound sane?) I started implementing the first reporter, and got stuck at an attempt to add structured something (so far it's just a string tag for simplicity) to the end of the message. I'm following tags.ml test file as an example, and at line 33 kfprintf is curried with user-provided format string. If I add my %s at the end of format string, that won't work. I was unable to wrap it in a way that does what I want. Any pointers are greatly appreciated, there's not much info on format magic in OCaml around.

Also would be great to get a piece of advice on how to approach the json reporter...

Thanks in advance!

Lupus commented 5 years ago

Okay, looks like I managed to figure out the appending at the end using kasprintf and chaining one more continuation with original kfprintf from the example. Probably I can encode the resulting message into json the same way, would that be the recommended option for json logging though?

Lupus commented 5 years ago

Hm, I managed to wrap it up in json using the same approach. Sorry for the noise :)

hannesm commented 5 years ago

@Lupus in case it helps, I developed a syslog reporter for Logs at https://github.com/hannesm/logs-syslog which works nicely for me (not yet RFC5424 structured syslog for tags due to some missing bits and pieces in https://github.com/verbosemode/syslog-message).

Lupus commented 5 years ago

Cool, I'll take a look! Thanks.