A Jaeger tracing plugin for Cassandra

Update: Works and tested with Cassandra 3.11.6!

This plugin is based on A Zipkin tracing plugin for Cassandra which is licensed under Apache License 2.0.

Cassandra provides pluggable tracing starting from version 3.4. By adding a jar file to the Cassandra classpath and one JVM option, Cassandra's tracing can be replaced with Jaeger. It can even identify incoming Jaeger traces and add Cassandra's own internal tracing on to it.

How to use

• Make sure you have Maven and JDK installed on your machine

• Run following commands to build and place the jar

  # Cloning the repository
  git clone https://github.com/infracloudio/cassandra-jaeger-tracing.git
  cd cassandra-jaeger-tracing
  
  # Create a jar file
  mvn package
  cp target/cassandra-jaeger-tracing-*-jar-with-dependencies.jar $CASSANDRA_HOME/lib/

  Here, $CASSANDRA_HOME is the directory where Cassandra is installed

• Start Cassandra with,

  JVM_OPTS\
  ="-Dcassandra.custom_tracing_class=io.infracloud.cassandra.tracing.JaegerTracing" \
  cassandra

  or edit the jvm.options

By default jaeger-client-java sends the spans to localhost:6831 via UDP. This can be configured by setting environment variables, JAEGER_AGENT_HOST and JAEGER_AGENT_PORT. Refer Configuration via Environment for more information.


Background

See CASSANDRA-10392 for the patch to extend Cassandra's tracing that this project plugs into.

Troubleshooting

When this tracing is used instead of Cassandra's default tracing, any cqlsh statements run after enabling tracing with TRACING ON; are going to time out eventually giving

Unable to fetch query trace: Trace information was not available within …

This is because cqlsh is polling for tracing information in system_traces which isn't any longer being created. For the meantime an easy fix around this behaviour in cqlsh is to reduce Session.max_trace_wait down to 1 second.

Continuing parent traces

In order to continue a parent trace send the trace injected into custom_payload with the _trace_idkey. Default is uber-trace-id, but it can be changed through an environment variable. Inject it using HTTP_HEADERS TextMap codec with url encoding value of true.

Refer to your Cassandra driver documentation in order to figure out how to send custom_payload.

If you need a custom trace key, specify it in environment variable JAEGER_TRACE_KEY. Note that the default is uber-trace-id.