search

DataDog / kong-plugin-ddtrace

Datadog APM Plugin for Kong Gateway
Apache License 2.0
15 stars 7 forks source link

Correlate Span ID and Trace ID to Kong logs. #25

Closed mfischler closed 1 month ago

mfischler commented 6 months ago

Version of kong: 3.5.0 Platform: kubernetes ddtrace version: 0.1.2-1 Configuration Type: Specific services Configuration Details:

{ "instance_name": null, "enabled": true, "updated_at": 1704398691, "route": null, "created_at": 1704398691, "id": "8bba4f3d-7c08-5dc7-a4e7-e77d6cf58e12", "tags": null, "consumer": null, "config": { "resource_name_rule": null, "static_tags": null, "service_name": "{vault://env-vault/KONG_SERVICE_NAME}", "environment": "{vault://env-vault/KONG_ENVIRONMENT}", "initial_sample_rate": null, "agent_endpoint": "{vault://env-vault/KONG_DATADOG_AGENT_TRACING_ENDPOINT}", "initial_samples_per_second": 100 }, "service": null, "protocols": [ "grpc", "grpcs", "http", "https" ], "name": "ddtrace" }

Detailed description of the problem, and if known, the expected behavior.

Not Sure if possible, is there a way to correlate the Span ID and Trace ID to the logs collected from Kong?

Validation:

Kong logs would include the tags "span.id" and "trace.id" generated by the Datadog Trace plugin.

dmehala commented 5 months ago

Hi @mfischler,

Thank you for reporting this issue. Our team is currently reviewing the details, and we'll work on addressing it as soon as possible.

To further enhance the visibility of your request and for future feature suggestions, we encourage you to submit them through our dedicated support platform at Datadog Support Portal.

mfischler commented 3 months ago

Closed a feature implemented using environment variables. Updated technique provided by Datadog.

danielpoonwj commented 1 month ago

can we store the trace id in ngx.ctx? then it can be up to the users to implement the log injection depending on their use case.

if you're using the file-log plugin, you can use custom_fields_by_lua to access the value in ngx.ctx and write it to dd.trace_id in the JSON log

dmehala commented 1 month ago

If ngx.ctx is shared between plugins, then it can be used to share traceID and spanID between plugins, thus making it available for file-log plugin.

On another note, I was able to correlate traces with logs by setting an NGINX variable. However, that solution would requires to manually update the log format which is not that straightforward for Kong.

danielpoonwj commented 1 month ago

If ngx.ctx is shared between plugins, then it can be used to share traceID and spanID between plugins, thus making it available for file-log plugin.

On another note, I was able to correlate traces with logs by setting an NGINX variable. However, that solution would requires to manually update the log format which is not that straightforward for Kong.

yup i think ngx.ctx, ngx.var or kong.ctx.shared can all be potentially used for exposing the trace id outside the plugin.

can we consider setting it on any of those places and documenting it for others to use? we actually maintain a private internal fork just for this but would love to use the upstream if this was implemented

dmehala commented 1 month ago

I opened #61. However, I need to play a bit with log correlation to know if the TraceID and SpanID is enough. Hopefully that will be merged and released by tomorrow. Thank you!