How can I integrate trace with log?

[tiendq]

Currently I see this in the documentation :)

Coming Soon. Reach out to the Datadog support team to learn more.

https://docs.datadoghq.com/tracing/advanced/connect_logs_and_traces/?tab=c

[cgilmour]

At the moment, the C++ tracer only exposes the opentracing interface, and not the inner workings and details of the tracer. This means it's not possible to extract the necessary data from the span (trace and span ID) and put them into log records.

Also the Log method that opentracing provides has not been implemented yet. That'd require additional changes outside of the tracer.

For better understanding, which of these approaches were you interested in? If you need more detail to answer, let me know - happy to explain more.

[tiendq]

@cgilmour If you look at detail view of a tracing on the DD, there is a Logs tab where it allows us to see tracing's related log entries. They are linked together via trace ID/span ID and I haven't seen how to do it with this library, no example provide on above link too.

Thanks,

[cgilmour]

Yup, understood. Right now, it's not possible.

Generally, the way that works is the application produces logs that include the trace ID and span ID. However, your code will only see the opentracing abstraction (opentracing::v2::Span), with no method to get the trace ID and span ID from them.

I don't have a solution for this yet.

[tomthomson]

If I understood correctly, this issue needs to be addressed first in opentracing-cpp to add support for retrieval of spanID/traceID in the opentracing::Span abstraction. I opened a request in opentracing-cpp to address this issue -> https://github.com/opentracing/opentracing-cpp/issues/112

[tomthomson]

2 virtual functions for returning TraceID and SpanID have now been added to the span interface in opentracing-cpp -> https://github.com/opentracing/opentracing-cpp/pull/114/files With the next release of opentracing-cpp these functions can then be implemented in dd-opentracing-cpp.

[tomthomson]

opentracing-cpp v1.6.0 has now been released: https://github.com/opentracing/opentracing-cpp/releases/tag/v1.6.0 -> including getters for spanID/traceID for a span

[cgilmour]

Thanks @tomthomson! Just FYI I got an alert for the release and already prepared some code changes to use it. They're on a branch for now, and we'll find a way to get that merged in and available in a release. https://github.com/DataDog/dd-opentracing-cpp/tree/cgilmour/opentracing-1.6.0

It might take a little while because for some downstream projects (eg: envoy, nginx-opentracing, etc) it'll need a similar update from other tracers they support before they can update to opentracing 1.6.0

[tomthomson]

Thanks for the quick adaption

[juniorz]

What about NGINX integration (via nginx-opentracing)?

nginx-opentracing makes $opentracing_context_name available to get a value of the active span context. What are the names of the values added by dd-opentracing-cpp to the span that contains the span_id and trace_id?

I have tried $opentracing_context_x_datadog_trace_id for trace_id but could not find the equivalent for span_id.

Use case here is integrating nginx logs with APM traces.

[alnr]

Is there an update on this?

[sergiodelacruz]

Is there any recent progress on this issue? We need it pretty badly.

[cgilmour]

@juniorz @alnr @sergiodelacruz see below and let me know if that covers what you need for integrating nginx tracing and logs.

For NGINX, the trace and span IDs are available via the special variables:

• $opentracing_context_x_datadog_trace_id
• $opentracing_context_x_datadog_parent_id

Although it says "parent" there, it's actually the current span ID. These values are taken from propagation headers for distributed tracing. When datadog tracing injects those headers, it uses x-datadog-parent-id to represent the span ID.

There are extra steps for those values to be picked up by NGINX logs. In addition to configuring NGINX logs collection by the agent, a new pipeline needs to be created to understand a log format that includes trace and span IDs. This is easier after traces are already being collected and the NGINX integration has already been added. It should show up in the list of pipelines.

First, clone the Nginx pipeline. This will create a new pipeline with the same name and disable the original one. Expand the new one and edit the Grok Parser. Between access.common and access.combined, add a new rule

access.traced %{access.common} "%{_referer}" "%{_user_agent}"( "%{_x_forwarded_for}")( "%{_trace_id}" "%{_span_id}")?

Expand Advanced Settings and add two new Helper Rules:

_trace_id %{word:trace_id}
_span_id %{word:span_id}

Save the rule changes.

Next, use Add Processor, with these settings:

• Processor type: Trace Id Remapper
• trace id attribute: trace_id
• name: Set the Trace ID attribute

The exact settings for the Grok Parser depend on the precise value of your nginx setup's log_format. This example is based on these log format settings.

    log_format with_trace_id '$remote_addr - $http_x_forwarded_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for" '
        '"$opentracing_context_x_datadog_trace_id" "$opentracing_context_x_datadog_parent_id"';

    access_log /var/log/nginx/access.log with_trace_id;

[sergiodelacruz]

Thanks for your response, @cgilmour. I'm still trying to wrap my head around it. What does NGINX has to do with this?

Our service is running in a Kubernetes pod and the DataDog agent picks up its stdout for the logs. As far as I can tell, there's no NGINX anywhere. We do have have the dd-opentracing-cpp library sending traces to the DataDog agents through HTTPS by setting up env vars like DD_TRACE_AGENT_PORT and the like.

Can you clarify how your latest comment would be relevant to our use case?

[cgilmour]

Sorry for the confusion @sergiodelacruz , other comments had referenced nginx which did have a workaround already available.

In your case, is it a C++ service using opentracing-cpp and dd-opentracing-cpp directly, and you want your logs from that to connect with traces? It can be done with opentracing v1.5.1, but is a bit cumbersome. If that's what you need for the short term, let me know and I'll write up the details.

[sergiodelacruz]

Thanks, @cgilmour for your prompt reply.

In your case, is it a C++ service using opentracing-cpp and dd-opentracing-cpp directly, and you want your logs from that to connect with traces?

Yes, that is correct. I'm using your cgilmour/opentracing-1.6.0 branch that integrates with opentracing-cpp v1.6 but the tracer.log() functions there are empty 😕

If I'm understanding correctly from DataDog's documentation, all that is needed is a log entry in JSON with the following there:

"dd": {
  "trace_id": "<the trace ID>",
  "span_id": "<the span ID>"
}

which is kinda hard to provide in the client code since IIRC those IDs are private to the Tracer class. Am I missing something here?

[cgilmour]

Great, I can prepare an end-to-end example for that.

The opentracing Log() functions are for adding event data to the span that gets reported. Some tracing systems, including Datadog APM, don't support that, so they are empty in this project's implementation. It's a confusing name, because it sounds like it'd be for producing application logs.

In earlier opentracing versions, there were no methods to get trace and span IDs. They existed but hidden by the abstraction layers of opentracing. In v1.6.0 they are available, eg:

    auto& ctx = span->context();
    auto trace_id = ctx.ToTraceID();
    auto span_id = ctx.ToSpanID();
    // do things with trace_id and span_id

I'll make sure an end-to-end example covers both the usage and the format(s) you can use.

[meme]

Hi all, is there an example available for connecting logs to traces in C++? Thanks.

[perseoGI]

Hi there! Im trying to send logs in correlation with a trace/span

auto jobSpan = m_tracer->StartSpan(taskRegister.jobType);
jobSpan->Log({ {"error", "this is a test"} });
jobSpan->SetTag("tag test", 42);
jobSpan->Finish();

Trace is being sent to datadog but I can't see any log in the Logs tab. I'm not sure if this is the way to go. Does Log method send a log to datadog? Or do I have to use another library in order to send the logs and relate them with trace_id and span_id?

Thanks in advance :smiley:

[surya-gopuff]

Hi,

IS the end-to-end example of combining logs with the traces using the trace and log ids there yet? Could you share the link in here?

Thanks in advance