Manual context propagation across microservices behind Pub/Sub system

[AnirudhaGohokar]

Hi ,

I want to propogate traceid context from one NodeJS microservice to another NodeJS microservice. Currently this is the architecture:

MicroserviceA >>> KAFKA >>> MicroserviceB

After using code as mentioned here using W3CTraceContextPropagator am unable to get the span under the same traceid.

Is this even possible with Signoz? Do you have a sample application for same?

This is code am using currently to propogate context via KAFKA in MicroserviceA

import {W3CTraceContextPropagator} from "@opentelemetry/core";
import {
    defaultTextMapGetter,
    defaultTextMapSetter,
    ROOT_CONTEXT,
    trace,
} from "@opentelemetry/api";

const parentSpan = tracer.startSpan('main');

// In MicroserviceA
const propagator = new W3CTraceContextPropagator();
let carrier = {};

propagator.inject(
    trace.setSpanContext(ROOT_CONTEXT, parentSpan.spanContext()),
    carrier,
    defaultTextMapSetter
);
console.log("carrier", carrier); // sending traceId via kafka

In downstream MicroserviceB am then extracting the parent context

const parentCtx = propagator.extract(ROOT_CONTEXT, carrier, defaultTextMapGetter);
const childSpan = tracer.startSpan("child", undefined, parentCtx);

[pranay01]

@AnirudhaGohokar Which opentelemetry related libraries are you using in your NodeJS application? And how is your NodeJS application connecting with Kafka?

[AnirudhaGohokar]

Am using below temetry libraries:

    "@opentelemetry/api": "^1.0.4",
    "@opentelemetry/auto-instrumentations-node": "^0.28.0",
    "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
    "@opentelemetry/instrumentation": "^0.29.2",
    "@opentelemetry/instrumentation-express": "^0.29.0",
    "@opentelemetry/instrumentation-http": "^0.29.2",
    "@opentelemetry/propagator-ot-trace": "^0.26.0",
    "@opentelemetry/resources": "^1.3.1",
    "@opentelemetry/sdk-node": "^0.27.0",
    "@opentelemetry/sdk-trace-node": "^1.3.1",
    "@opentelemetry/semantic-conventions": "^1.3.1",
    "@opentelemetry/tracing": "^0.24.0",

For connecting to kafka am using node-rdkafka

[AnirudhaGohokar]

This is the tracer.js file content which am invoking as node -r tracing.js index.js

'use strict'
const { OTLPTraceExporter } = require('@opentelemetry/exporter-otlp-grpc');
// configure the SDK to export telemetry data to the console
// enable all auto-instrumentations from the meta package
const traceExporter = new OTLPTraceExporter();
const {
    BasicTracerProvider,
    ConsoleSpanExporter,
    SimpleSpanProcessor,
    BatchSpanProcessor,
  } = require('@opentelemetry/tracing')
  const { Resource } = require('@opentelemetry/resources')
  const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions')
  const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express')
  const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http')
  const { registerInstrumentations } = require('@opentelemetry/instrumentation')
  const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node')
  const { OTTracePropagator } = require('@opentelemetry/propagator-ot-trace')
  const { W3CTraceContextPropagator } =  require("@opentelemetry/core");

  const init = (serviceName, environment) => {

    const provider = new NodeTracerProvider({
      resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: serviceName, // Service name that showuld be listed in jaeger ui
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: environment,
      }),
    })

    //provider.addSpanProcessor(new SimpleSpanProcessor(exporter))

    // Use the BatchSpanProcessor to export spans in batches in order to more efficiently use resources.
    provider.addSpanProcessor(new BatchSpanProcessor(traceExporter))

    // Enable to see the spans printed in the console by the ConsoleSpanExporter
    // provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter())) 

    provider.register({ propagator: new W3CTraceContextPropagator() })

    registerInstrumentations({
      instrumentations: [new ExpressInstrumentation(), new HttpInstrumentation()],
    })
    global.tProvider=provider;
    const tracer = provider.getTracer(serviceName)
    return tracer
  }

  if(!global.tracer)
  {
    global.tracer = init('my-service-name','development');
    console.log('Tracer Initialized');
  }else{
    console.log('Tracer already initialized');
  }

Now when publishing from MicroserviceA via KAFKA am doing this to get traceID:

const api = require('@opentelemetry/api');
const { W3CTraceContextPropagator } =  require("@opentelemetry/core");
....
const parentSpan = global.tracer.startSpan('parent_span');
const propagator = new W3CTraceContextPropagator();
let carrier = {};
propagator.inject(
        api.trace.setSpanContext(api.ROOT_CONTEXT, parentSpan.spanContext()),
        carrier,
        api.defaultTextMapSetter
 );
console.log("carrier", carrier);//sending this traceId via kafka

Now when consuming via Kafka in MicroserB am doing this:

const api = require('@opentelemetry/api');
const { W3CTraceContextPropagator } =  require("@opentelemetry/core");
....
const propagator = new W3CTraceContextPropagator();
const parentCtx = propagator.extract(api.ROOT_CONTEXT, <<carrier>>, api.defaultTextMapGetter);
const childSpan = global.tracer.startSpan("child", undefined, parentCtx);

[srikanthccv]

After using code as mentioned https://github.com/open-telemetry/opentelemetry-js/issues/2458#issuecomment-913849614 using W3CTraceContextPropagator am unable to get the span under the same traceid. Is this even possible with Signoz? Do you have a sample application for same?

@AnirudhaGohokar The trace context propagation is entirely the client instrumentations work. How are you transporting the W3C trace parent value to downstream consumer service? If you log the carrier do you the see the carrier containing traceparent header in consumer? It would look something like this

traceparent: '00-b833af7bc3e35afeaf3955a922273c9d-017ea7b504ba9d57-01'

[buicuong1303]

I have the same issue. I got traceparent: '00-b833af7bc3e35afeaf3955a922273c9d-017ea7b504ba9d57-01'. But I can not extract the parent context in service B

[srikanthccv]

But I can not extract the parent context in service B

What is the issue you are facing while extracting the context? Is the propagator (from SDK) not working or something else?

[buicuong1303]

I think the propagator is not working. I use gRPC to communicate service A and B. I don't know how to propagate the context between 2 services

[srikanthccv]

Which propagator are you using? All the propagators provided by SDK work. You could still propagate span context even when you are using gRPC with metadata headers. You first serialize the context to string and then extract context out of string to be used by downstream service. It would look something like this

const parentSpan = global.tracer.startSpan('main');

// In upstream service
const propagator = new W3CTraceContextPropagator();
let carrier = {};

propagator.inject(
    trace.setSpanContext(ROOT_CONTEXT, parentSpan.spanContext()),
    carrier,
    defaultTextMapSetter
);
console.log("carrier", carrier); // transport this carrier info to other service via headers or some other way

// In downstream service

const parentCtx = propagator.extract(ROOT_CONTEXT, carrier, defaultTextMapGetter);

const childSpan = global.tracer.startSpan("child", undefined, parentCtx);

[buicuong1303]

Thank you. I will try again.

[mahipatsinghtellius]

Hi!

I am too not able to extract it, any update?

[srikanthccv]

@mahipatsinghtellius Are you still facing the issue? What is the propagator you are using and can you clarify where you are facing the issue?