Ktor OpenTracing Instrumentation

Library of Ktor features for OpenTracing instrumentation of HTTP servers and clients.

Usage

Server Spans

Install the OpenTracingServer feature as follows in a module:

install(OpenTracingServer)

The feature uses the tracer registered in GlobalTracer, which uses the ThreadContextElementScopeManager. This is needed to propagate the tracing context in the coroutine context of the calls. For example, you can instantiate and register a Jaeger tracer in the module before the call to install as follows:

val tracer: Tracer = config.tracerBuilder
    .withScopeManager(ThreadContextElementScopeManager())
    .build()

GlobalTracer.registerIfAbsent(tracer)

At this stage, the application will be creating a single span for the duration of the request. If the incoming request has tracing context in its HTTP headers, then the span will be a child of the one in that context. Otherwise, the feature will start a new trace.

Individual code blocks

To get a more detailed view of requests, we might want to instrument individual code blocks as child spans. We could start a new child span using the tracer instance directly, however this would be too intrusive and verbose. Instead, we can use the span inline function as follows.

class UserRepository {
    fun getUser(id: UUID): User = span("<operation-name>") {
        setTag("UserId", id)

        ... database call ...

        return user
    }
}

span is passed an operation name and an anonymous lambda, which has the Span as a receiver object. This means that you can call setTag, log, getBaggageItem (or any method on the Span interface).

Concurrency with async

Concurrent operations using async can break in-process context propagation which uses coroutine context, leading to spans with incorrect parents. To solve this issue, replace the calls to async with asyncTraced. This will pass the correct tracing context to the new coroutines.

val scrapeResults = urls.map { url -> 
    asyncTraced { 
        httpClient.get(url)
    }
    .awaitAll()
}

Underneath the hood, asyncTraced is adding the current tracing context to the coroutine context using a call to tracingContext(). You can add it yourself by calling async(tracingContext()). To launch a new coroutine with the tracing context, call launchTraced.

Client Spans

If your application calls another service using the Ktor HTTP client, you can install the OpenTracingClient feature on the client to create client spans:

install(OpenTracingClient)

The outgoing HTTP headers from this client will contain the trace context of the client span. This allows the service that is called to create child spans of this client span.

We recommend using this feature in a server that has OpenTracingServer installed.

Configuration

Filter Requests

Your application might be serving static content (such as k8s probes), for which you do not to create traces. You can filter these out as follows:

install(OpenTracingServer) {
    filter { call -> call.request.path().startsWith("/_probes") }
}

Tag Spans

It is also possible to configure tags to be added to each span in a trace. For example to add the thread name and a correlation id:

install(OpenTracingServer) {
    addTag("threadName") { Thread.currentThread().name }
    addTag("correlationId") { MDC.get("correlationId") }
}

Installation

From Maven Central.

Maven

Add the following dependency to your pom.xml:

<dependency>
 <groupId>com.zopa</groupId>
 <artifactId>ktor-opentracing</artifactId>
 <version>VERSION_NUMBER</version>
</dependency>

Gradle

Add the following to your dependencies in your build.gradle

implementation "com.zopa:ktor-opentracing:VERSION_NUMBER"

Examples

• For a simple example of ktor app instrumented with OpenTracing, see ktor-opentracing-example. This app uses span names passed explicitly to the span inline function.

• For automatic span naming using the class and method name, see ktor-opentracing-span-naming-demo.

Related Projects

• For Ktor services using kotlin-logging, you can use kotlin-logging-opentracing-decorator to enrich your spans with logs.
• If you are using Exposed, you can use Exposed-OpenTracing to instrument database transactions.