Migrate to Kamon Metrics

[jasongilanfarr]

Particularly beneficial:

• HdrHistogram by default
• Built-in Akka Metrics
• TraceContext

I would really like to be able to attach a TraceID to every request and be able to see all log statements (whether via actors, futures, etc) related to a request have an MDC traceid associated with them.

As a secondary benefit, we could also expose a way to enable the different metric writers for customers (e.g. datadog, etc).

[janisz]

I'm interested in this feature since it's related to #4087. For me Kamon will give me the opportunity to set PathId/AppId/TaskId in MDC to trace actions taken to serve particular HTTP request or some internal actions like health checking. I was thinking about using StructuredArguments to do it but using Kamon MDC could be better. Trace Token is nice to have too.

What are the steps that need to be done? I can help with implementation but need some guidance.

[jasongilanfarr]

@janisz Sorry I missed your update. We are planning on adding MDC to HTTP soon- I may do this as soon as next week. The basic start would be migrating from MetricRegistry/Metrics and using Kamon directly. I'll look a bit more today and see if I can elaborate on other requirements.

[janisz]

@jasongilanfarr I started working on structured logging. It looks like we cannot use StructuredArguments when akka.actor.ActorLogging is used but it works perfectly to normal (non Akka) logs. It turns out TraceLocal.store works only when context was enabled with Tracer.withNewContext.

I think StructuredArguments could be used instead of MDC in non Akka Actors code. Intorducing them require only minimal changes to code, only in log statements. Example

-    log.info(s"Starting Marathon ${BuildInfo.version}/${BuildInfo.buildref} with ${args.mkString(" ")}")
+    log.info(
+      "Starting Marathon {}/{} with {}",
+      v("version", BuildInfo.version),
+      v("buildref", BuildInfo.buildref),
+      v("args", args.mkString(" "))
+    )

Default logs looks same but in JSON logs, version, buldref and args are added as separated fields. To achieve same effect with MDC we will need for wrap each log statement with MDC.put and remember about clearing it.

Kmon logs works with Akka but it require additional dependencies and some boilerplate code. Each log statement need to be wrapt with withMdc and fields need to be added to context. So it's definitely not a small change and I think we can introduce it once we add Kmon metrics and setup context.

If it's ok with you I can work on StructuredArguments in HTTP instead of MDC. Here are my tires on structured logging.

[jasongilanfarr]

Preview is here https://phabricator.mesosphere.com/D363 - still a decent amount todo:

• [ ] Enable on /v2/metrics
• [ ] Write a custom Json formatter
• [ ] add back in timings/tracing for the API handlers
• [ ] Attempt to preserve some semblance of compatibility

Optionals:

• [ ] Throw chaos/guice out the window in favor of jersey+jetty (until we can do akka http only).

[jasongilanfarr]

@janisz please take a look if you're interested... https://phabricator.mesosphere.com/D363 I didn't do anything with TraceContext yet as we're going to move straight over to Akka Http soon, but all the base metrics are in.