search

DataDog / dd-trace-dotnet

.NET Client Library for Datadog APM
https://docs.datadoghq.com/tracing/
Apache License 2.0
437 stars 137 forks source link

CloudWatch log enrichment for Lambda invocations no longer works after upgrading to version 3.2.0 #6006

Closed KevinD-87 closed 3 weeks ago

KevinD-87 commented 3 weeks ago

Describe the bug I'm using the latest versions of the Datadog Lambda extension (v64) and Datadog .NET Tracer AWS Layer (v15 - dd-trace-dotnet 2.46.0). My application is running on .NET 8.0 and contained a reference to Datadog.Trace 2.57.0, which allowed me to enrich the logs that are written to CloudWatch with the dd_trace_id, dd_span_id, etc. After upgrading Datadog.Trace to version 3.2.0, this enrichment no longer works. The following environment variables are configured on my Lambda function:

    DD_TRACE_ENABLED                 = true
    DD_TRACE_MANAGED_SERVICES        = true
    DD_LOGS_INJECTION                = true
    DD_TRACE_DEBUG                   = false
    DD_CAPTURE_LAMBDA_PAYLOAD        = false
    DD_SERVERLESS_LOGS_ENABLED       = false

I've tried adding the following environment variables, as described in the single step instrumentation documentation, which was linked via the 'Automatic Instrumenation' page in the Migration documentation, since I'm suspecting an incompatibility between the version of dd-trace-dotnet used by the Lambda layer and the Datadog.Trace package, but this doesn't work either:

    DD_APM_INSTRUMENTATION_ENABLED   = "host"
    DD_APM_INSTRUMENTATION_LIBRARIES = "dotnet:2.46.0"

To Reproduce Steps to reproduce the behavior: see description

Expected behavior CloudWatch logs are enriched with Datadog context information like the dd_trace_id, dd_span_id, etc.

Runtime environment (please complete the following information):

andrewlock commented 3 weeks ago

Hi @KevinD-87, yes looks like an incompatibility between the versions. You should always keep the version of your Datadog.Trace NuGet package in-sync with the version used in the AWS Layer. The Lambda team are working on producing a layer with a newer version of the library, but in the mean time you should stay on 2.x.

FYI, one of the reasons for the breaking changes in the v3 version of the tracer was to reduce the coupling between the NuGet and automatic instrumentation versions, so that it becomes less important to keep them perfectly in sync (as long as both are on v3!)

KevinD-87 commented 3 weeks ago

Hi @andrewlock, thanks for the quick confirmation! I'll keep an eye on the releases of the AWS layer.

duncanista commented 3 weeks ago

Hey @KevinD-87,

Just chiming in from the Serverless side, we should have a new Layer release very soon this week!

KevinD-87 commented 3 weeks ago

Hey @KevinD-87,

Just chiming in from the Serverless side, we should have a new Layer release very soon this week!

That's great news. Thanks!