Question about logging/MDC fields

[StephenGoodall]

@manikmagar sorry to bug you again so soon, this isn't really an issue, more of a question if thats ok?

Will this module add the OTel related data into the MDC fields for the existing logging? I'm trying to set this up in conjunction with https://github.com/michaljonko/dynatrace-log4j2-appender and it mentions that the ${otel:trace_id} is dependent on having the OpenTelemetry Instrumentation for Java in place - I guess my question is whether your module incorporates this, or would I need to add that in as a separate dependency to get those fields coming through?

Apologies if this isn't really related to your module - still getting my head around it........I'm a total mulesoft noob who's been tasked with getting the logs and traces into dynatrace 😄

edit: this could explain it? https://help.mulesoft.com/s/ideas#0872T000000XbzdQAC

[manikmagar]

Hi @StephenGoodall , no worries.

I think you can use Tracing module - https://docs.mulesoft.com/tracing-module/1.0/tracing-module-logging. You may need to disable CloudHub logging to allow using application's log4j. You can use vars.OTEL_CONTEXT.traceId in your flow to set in the logging variable or wherever you need.

[adesjardin]

@StephenGoodall - What @manikmagar has described is what we have been doing currently at the beginning of requests/flows. We do want to try and add that MDC variable automatically in the future though.

[StephenGoodall]

thanks @manikmagar @adesjardin - I went ahead and made those changes, then realised we are on Mule 4.3.0 and Tracing needs 4.4.0 🤦‍♂️ so I've put it on the back burner for now

Thanks for the advice and the link!

[StephenGoodall]

Is there any way to get the span ID aswell, or is that not currently captured in the OTEL_CONTEXT object?

The TraceId is working great now that we've upgraded to mule 4.4 and added the tracing module!

[manikmagar]

@StephenGoodall Glad to know that traceid is working great.

We don't capture Span Id separately in the context object. I don't know the intention of looking for span id but You can take a look at Context extraction. Check the traceparent header for W3 Trace context it can show you the parent span id.

[StephenGoodall]

Thanks again @manikmagar I'll have a look into the extraction, maybe I can use OTEL_TRACE_CONTEXT.traceparent value then split it using dataweaver to get that span ID value out of that string.

Just for context, I'm trying to get that span ID so I can get dynatrace to link the logs to the span, it works well with the trace ID but it would be great to be able to drill down into the logs at a span level.

The distributed traces section of dynatrace shows me the span ID, which I think OTel is populating (thanks to your module) in the OTLP export. Dynatrace just also needs to have a specific "dt.span_id" key/value pair in the logs to correlate them to the span I think.

If I get it working, I'll report back in case anyone else might find it useful in the future!

[StephenGoodall]

The Dataweave option worked! 🥳 PS. add the default value on the end otherwise bad things happen.......

Details below for reference, if anyone else searches for this.

In my flows I added these:

        <tracing:set-logging-variable  variableName="mule_trace_id" value="#[vars.OTEL_TRACE_CONTEXT.traceId default '' as String]"/>
        <tracing:set-logging-variable  variableName="mule_span_id" value="#[((vars.OTEL_TRACE_CONTEXT.traceparent default '' as String splitBy '-')[2]) default '']"/>

In my log4j I added these:

            <Property name="dt.trace_id">${ctx:mule_trace_id}</Property>
            <Property name="dt.span_id">${ctx:mule_span_id}</Property>

I still need to dig into it a bit more to check on whether that is the right value to pull out - I feel like this is getting me the parent span not the "current" span - but its definitely progress!

[manikmagar]

Glad to see the progress. Thank you for sharing this here.

I still need to dig into it a bit more to check on whether that is the correct value to pull out - I feel like this is getting me the parent span, not the "current" span - but it's definitely progress!

To confirm, traceparent will always include the parent span id.