Modify the log output format

[KuanYa]

I want to modify the original log output format, and the adjusted format is JSON, what should I do?

[vadorovsky]

There is no way to configure output format for now. What kind of format would you be interested in?

It's a feature request. And it could be a good occasion to switch from log to tracing.

[bcelenza]

Love this idea and started playing around with it in a fork.

One gotcha I ran into was how to handle the "log level" of a threat in a way that would work for both structured and unstructured logging. The current implementation constructs a log-like line with the THREAT or EVENT level, but this makes it separate from the overall logging framework. Neither the core log or tracing crates support a custom log level, so it makes sense why it was done this way originally.

There are a couple options I can think of to get around this (disclaimer: I'm not a tracing crate expert):

1. Emit threats with a defined log level (e.g., WARN), and a separate field name/value to indicate whether it's a threat or event. This would allow the threats and events to be filterable in a tracing subscriber layer, and we could make decisions on how to log them out to console as a result.
2. Decouple logging of the pulsar daemon from the reporting/output of threats and events such that they are pushed to different places (and the threat reporting does not use the logging crate). For example, standard daemon logs could be reported to stderr and threats/events could be reported to stdout. This has the added benefit of being able to filter or focus on what an operator cares about.

Option (1) allows the threats and events to fit within any logging crate and use its subsequent output features (i.e., structured vs unstructured) at the cost of them potentially being lost in a sea of other logs at the same level. Option (2) frees us from having to try and shoehorn threat reporting into a logging framework, but would require us to separately implement structured vs. unstructured reporting.

Personally, I like option (2), as the code to output threats and events in various formats wouldn't be terribly onerous to implement and maintain. I also like that it separates diagnostic logging from threat reporting, but that could be achieved in option (1) with tracing layers.

[banditopazzo]

Hi,

I agree with you on Option 2, having separate application logging and threat logging, because they are different things and shouldn't be mixed.

The log and tracing crates are meant for application logging, connected in general to the stderr.

The pulsar logger instead logs threats but to the stdout.

The output format of the pulsar logger should be something you can select in the options, independent from the application logger. The Event struct it's already Serialize, so we need just the format option in the logger config and the logic to handle the serialization, JSON in your case.

I'm not a tracing expert neither and honestly I don't know exactly how layers work, but again I prefer having the logger module independent from the app.

About the switch from log to tracing for application logging, it's a different matter, we could do that too, just in a different PR from the logger changes.

[bcelenza]

Thanks @banditopazzo, I didn't catch that threat logging had already been split to stdout. Posted #277 to gather some feedback on the initial approach.

[banditopazzo]

Fixed by #277