search

DataDog / datadog-lambda-go

The Datadog AWS Lambda package for Go
Apache License 2.0
59 stars 40 forks source link

Enhanced metrics always flushed to logs #94

Closed ynguyensegment closed 2 years ago

ynguyensegment commented 2 years ago

Expected Behavior

According to the documentation for the Datadog Lambda Extension, enhanced metrics should be sent to Datadog by the extension when it is enabled, rather than going through CloudWatch logs.

serverless_monitoring_installation_instructions 7575f746ca248d623aee1ee646325460

Actual Behavior

Enhanced metrics are being force pushed to logs while custom metrics are not: https://github.com/DataDog/datadog-lambda-go/blob/757bd4fa779014c69fc1659ce558b0b960269f22/internal/metrics/listener.go#L202

We are able to receive all custom metrics in our dashboards but no enhanced metrics are showing up.

An example from a lambda function running with the Datadog Lambda Extension and DD_FLUSH_TO_LOG = false

// enhanced metrics being flushed to logs and do not show up
2021/09/29 23:20:58 {"status":"debug","message":"datadog: sending metric via log forwarder"}
{"m":"aws.lambda.enhanced.invocations","v":1,"e":1632957658,"t":["functionname:***","region:eu-west-1","account_id:***","memorysize:128","cold_start:false","datadog_lambda:v1.3.0","resource:***","dd_lambda_layer:datadog-go1.15.15"]}

// custom metrics are not flushed to logs 
2021/09/29 23:20:58 {"status":"debug","message":"datadog: adding metric \"***\", with value 1.000000"}

Is this a bug in the code or are we still expected to run a Datadog Forwarder along side the Datadog Lambda extension in order to get enhanced metrics?

Steps to Reproduce the Problem

  1. Run a lambda function with DataDog Lambda Extension and enhanced metrics enabled
  2. Check that only non enhanced metrics are showing up in datadog

Specifications

ynguyensegment commented 2 years ago

This turns out to be a configuration issue on our side.