xray for tracing kafka messages

[continuum-aditya-pingale]

Hello,

I am trying to figure out how can use xray to instrument services that produce/consume kafka messages and have end-to-end tracing across event-driven flows. This seems to supported in Java as illustrated by this post - https://aws.amazon.com/blogs/opensource/tracing-performance-messaging-applications-kafka-aws-x-ray/ However, we are using golang to develop our kafka services. So I have been trying to figure out how to achieve this using the xray golang sdk. I haven't been able to figure out if my implementation is wrong or are there limitations with Xray or xray-go API.

topic := "xray"

ctx, seg := xray.BeginSegment(context.Background(), "service")
        defer seg.Close(nil)

ctxx, subSeg := xray.BeginSubsegment(ctx, "domain-event")
//ctx := context.Background()
    err = xray.Capture(ctxx, "event.ConsumeClaim", func(ctx1 context.Context) (err error) {
        msg := &sarama.ProducerMessage{
            Topic: topic,
            Value: sarama.StringEncoder("my_message_to_kafka"),
        }

        partition, offset, err := producer.SendMessage(msg)
        if err != nil {
            panic(err)
        }
        fmt.Printf("Message is stored in topic(%s)/partition(%d)/offset(%d)\n", topic, partition, offset)

        return
    })

I can see something like this in aws console

Not exactly able to trace kafka message. Am i doing something wrong here? @luluzhao You seem to be working on the similar issue.

[willarmiros]

Hi @continuum-aditya-pingale,

It appears you're on the right track. You seem to be instrumenting a Kafka producer. To see the end-to-end trace of your producer -> consumer on the X-Ray service map, you must also instrument the consumer code by parsing out the Trace ID, segment ID, and sampling decision from the trace header and beginning a segment with that data as described in that blog post.

Another observation is that you're creating a subsegment domain-event and using the xray.Capture() API. The capture API automatically creates a subsegment, so you probably do not need to manually create the domain-event subsegment. If you choose to keep both, then you must include a subSeg.close() at some point in your function to close the domain-event subsegment, since you are not doing so at the moment.

With those suggestions, your instrumentation of the producer should be good. If you have further questions about instrumenting the consumer we'd be happy to assist.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.