Concept to classify "Application logs" and access-logs

[IljaN]

Recently a few PRs got merged in reva in ocis to reduce log verbosity. During this process we realized that there might be a conceptual error as in that we want to sparate access-logs from application-logs to be able to filter based on interest.

A first idea would be to tag access-log lines so they can be conveniently filtered in your $LOGGING_SOLUTION interface. (grep, graylog, logstash...)

[butonic]

While reva and ocis have opentracing built in most of the logs currently go to stdout. We should move most log messages to trace spans.

From https://winderresearch.com/logging-vs-tracing-vs-monitoring/#logging

Logging [...] But the problem is that obtaining, transferring, storing and parsing logs is expensive. Because of this it is crucial to only log what is necessary; only logs that can be acted upon should be stored. Log only actionable information. [...]

Tracing A trace represents a single user’s journey through an entire stack of an application.

Metrics

For some higher level background: https://copyconstruct.medium.com/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e

For a more practical read with jaeger: https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941 (or as a video)

Follow these semantic recommendations: https://github.com/opentracing/specification/blob/master/semantic_conventions.md

Regarding metrics: https://medium.com/faun/how-to-monitor-the-sre-golden-signals-1391cadc7524

Opentracing can be used to collect metrics as well: https://www.hawkular.org/blog/2017/06/26/opentracing-appmetrics.html

Keep in mind that in production the tracing might be sampled, aka not every request is traced. Maybe only once per second per route or whatever sampling the tracing gods fancy.

More links: https://techbeacon.com/enterprise-it/monitoring-demystified-guide-logging-tracing-metrics

[IljaN]

Tough when no opentracing is available the system should imo fallback to stdout and dump everything there. Everything should be available in both worlds. No?

[labkode]

Tracing is a nice addition on top of logging, but logging is the foundation, it cannot be switched of in favour of tracing as traces are much more simple and lightweight. A tag can be added to access logs (grpc and http).