Add top level support for logging

[savaki]

In addition to metrics and traces, opencensus should also include support for logging. Tags are an incredibly useful feature and being able to share a common set of tags across metrics and logging is incredibly useful not to mention the ability to correlate logs with traces as referenced here:

https://github.com/census-instrumentation/opencensus-specs/issues/123

[codefromthecrypt]

can we clarify what the goal is here a little more specifically?

Are we looking at a generalized feature for log ingest that allows correlation (like stackdriver enhancer) this would be appender based I guess..

or is this a correlation with other logging library contexts (like MDC in log4j)?

[songy23]

/cc @g-easy @sebright

[savaki]

Using go as a frame of reference, what I'd love to see is a trace and tag aware logging interface that can be implemented by different providers.

We use opencensus-go internally and have written an internal library to handle this that we've found to be exceptionally useful. Being able to filter log records by a particular user or org make troubleshooting much easier. This is even more magical when coupled with the log tailing feature of Stackdriver.

To move the conversation forward, I put together a working experiment of what this library might look like, https://github.com/savaki/opencensus-go/tree/logging/log Like sl4j, this log package is an interface without an implementation. In go, I'd expect exporters would be written for logrus, zap, and other popular packages

In languages like Java, I would imagine we would want to make available exporters for common packages like sl4j or log4j

[codefromthecrypt]

Thanks for the info. so this sounds like a primary interface for logging, then. Ex user code would use it directly instead of another library. Would you expect this only to be the case in languages where there is no dominant logging library? Ex in java would you expect folks to use something like this instead of SLF4J or log4j2 etc? Or would you use an integration pattern there instead

(asking the more general question as this is a spec repo so seemingly cross language)

[savaki]

That's a good question.

As I understand it, one of the main value propositions of OpenCensus is that it provides standard interface for common operational operations like tracing and metrics across many languages and many service providers. When I write to the OpenCensus API, I have the promise of portability between the supported providers. It's reasonably easy to swap out Zipkin for Stackdriver for example.

The intent of this proposal is to extend that promise to logging. Disclaimer: I'm not very familiar with the OC Java implementation. That said, it seems that if we want to allow users to integrate logging an existing Java app with OC, there are a number of approaches.

Configuration Only In this approach, no code changes are required. All the magic would be done inside the SLF4J or log4j2 configuration using a custom appender like thing. I'm a little confused as to how this would work as I understand propagating ThreadLocal data over futures is still an outstanding issue. And if developers haven't changed their code and they don't have access to ThreadLocal data, I'm not sure how OC would get trace or tag data. Granted you could give a big disclaimer that this approach only works with non-async code, but that doesn't seem like a good approach.

Custom Marker In the case of SLF4J, it looks like the have this Marker thing that might be appropriated for OpenCensus use. I haven't used it myself so I'm only guessing based on the apidoc. In this case, where ever we want the addition trace/tag data, we'd need the developer to inject an OC aware Marker. And given there may be existing code, it seems reasonable to expect OC to have to support wrapping existing Markers. I also need extend SLF4J to support OC configurations so swapping providers becomes a little more difficult. And while this seems like it ties OC to SLF4J, given the wide adoption of these libraries, for Java, I understand something like this might be the way to go.

Primary Interface However, if the developer already has to modify the code where ever they would like to get this augmented data, why not provide a common interface and have them write to that? The would be no messy interaction between OC and existing marker like types. Swapping providers would be easier. The common interface could still write out to log4j.

I'm not a Java expert, so I certainly admit I could be missing something.

[codefromthecrypt]

I think there is an option 4 in java, which is to synchronize a logging context when applying the tracing one. That's what's in use by brave for example and doesn't require users to depend on instrumentation code. Note it is also needed as many can't or won't change other libraries like zookeeper's logging library etc.

https://github.com/openzipkin/brave/blob/master/context/slf4j/src/main/java/brave/context/slf4j/MDCCurrentTraceContext.java#L55

In practice, folks write custom plugins to propagate more data than IDs for example to the logging context, or data from other sources. Such plugins don't affect call sites.

There are also plugins for this at least in log4j2 maybe also in the new slf4j as well not sure.. https://logging.apache.org/log4j/2.x/manual/extending.html#Custom_ContextDataInjector

Note: I've never seen ContextDataInjector set outside the internals of log4j2 itself, but that doesn't mean it hasn't happened

[ptone]

Finding this issue in search of an open-census opinion on logging.

I think a point of value of open-census is a nice decoupling from platform implementation. So metrics instrumentation can work with multiple backends. The same could be applied to logging in way that logrus has done for golang (eg I wrote a stackdriver logging plugin for logrus - stackrus). But logrus is golang centric, while open-census is polyglot. If a large team wanted to adopt a standard logging framework, and then leave infrastructure implementation up to a platform team, open-census seems like a good project to provide that decoupling. (correlation to traces is a bonus, but different thing)

[otoolep]

CC @c24t @mtwo

[otoolep]

@c24t @mtwo -- see https://github.com/census-instrumentation/opencensus-specs/issues/130#issuecomment-484997460 above.

[mtwo]

@ptone this is something that we're very interested in, and @c24t is working on some prototypes in Python. We wrote a plan a few months ago describing our goals and current thinking - feel free to comment or suggest edits (open it, apply for comment access, I'll grant it).