Hi! I've recently become more interested in structured logging, and have looked into a few structured logging libraries.
You get amazing power when you dump the logs from all of your different systems and sources into a centralized log store, and can then view and analyze them as one whole.
What I've noticed though is that the various structured logging frameworks all save JSON log entries in similar, but slightly different schemas.
For example, for storing timestamps, this library uses the JSON key "time", while other libraries use "timestamp" or "at". Another area where libraries differ is in how they store log levels.
These small differences cause friction when analyzing the central logstore, which contains structured logs that have been collected from multiple systems/microservices.
For example, one service might tag warnings with the string "WARN" while another with the string "warning". So if I want to view only warnings, I need to take this difference into account and write a tricky "OR" filter expression. This may seem minor, but these small inconsistencies cause great pain.
I believe that these small differences between the various structured logging libraries exist not because of any strongly held opinions, but simply because mainstream structured and centralized logging is still relatively young, and so there is no standard or common consensus.
I think it would be very beneficial to everyone if we could all unite around a common format.
It contains a comparison between several structured logging libraries, summarizing the differences between them all. This can hopefully be a start to help us arrive at common ground.
I encourage the authors of this library (and anyone else who has an opinion) to participate in the discussion!
Hi! I've recently become more interested in structured logging, and have looked into a few structured logging libraries.
You get amazing power when you dump the logs from all of your different systems and sources into a centralized log store, and can then view and analyze them as one whole.
What I've noticed though is that the various structured logging frameworks all save JSON log entries in similar, but slightly different schemas.
For example, for storing timestamps, this library uses the JSON key "time", while other libraries use "timestamp" or "at". Another area where libraries differ is in how they store log levels.
These small differences cause friction when analyzing the central logstore, which contains structured logs that have been collected from multiple systems/microservices.
For example, one service might tag warnings with the string "WARN" while another with the string "warning". So if I want to view only warnings, I need to take this difference into account and write a tricky "OR" filter expression. This may seem minor, but these small inconsistencies cause great pain.
I believe that these small differences between the various structured logging libraries exist not because of any strongly held opinions, but simply because mainstream structured and centralized logging is still relatively young, and so there is no standard or common consensus.
I think it would be very beneficial to everyone if we could all unite around a common format.
To get things started, I have created a GitHub repository to centralize discussions here: https://github.com/bitc/structured-logging-schema
It contains a comparison between several structured logging libraries, summarizing the differences between them all. This can hopefully be a start to help us arrive at common ground.
I encourage the authors of this library (and anyone else who has an opinion) to participate in the discussion!
Discussion takes place in the issues for this repo: https://github.com/bitc/structured-logging-schema