search

BrunoBonacci / mulog

μ/log is a micro-logging library that logs events and data, not words!
https://cljdoc.org/d/com.brunobonacci/mulog/
Apache License 2.0
490 stars 48 forks source link

JSON-file publisher? #88

Closed emlyn closed 2 years ago

emlyn commented 2 years ago

It would be useful to be able to write logs to a file in json-lines format (one json object per line), as this can be directly loaded into a Spark Dataset (and probably other uses too). I couldn't find a built-in way to do this, is that the case or did I miss something?

I wrote a simple publisher to do it (based on simple-file-publisher, and using charred for JSON conversion), would it be worth opening a PR to add it? If so, where would be the best place to put it? Inside mulog-json?

BrunoBonacci commented 2 years ago

Hi,

This seems to be a common use case. I've added a quick implementation in a discussion topic (https://github.com/BrunoBonacci/mulog/discussions/86#discussioncomment-2552989)

I'm considering adding this as a native solution, but I will need to add a new module for it. However, the json published shouldn't depend directly on charred but should use the mulog-json API to-json for serialisation.

Please, give me a few hours to add the module skeleton, and then you should be able to make a PR with the JSON publisher.

emlyn commented 2 years ago

Sounds good, thanks!

BrunoBonacci commented 2 years ago

Hi, i've pushed the skeleton for the advance file publisher, you add the implementation and tests with your PR:

https://github.com/BrunoBonacci/mulog/blob/master/mulog-adv-file/src/com/brunobonacci/mulog/publishers/file_json.clj

many thanks

emlyn commented 2 years ago

Thanks! I'll try to get a PR done in the next day or two.