DD Lambda Extension - connection refused (even though the extension layer is added)

[HannesBBR]

Expected Behavior

We are trying to set up distributed tracing, where a Golang lambda is the entrypoint of our flow that we want to trace. We want to use the lambda extension, and are using the datadog-lambda-go package pointing to the maxday/support-tracing-with-extension branch of @maxday.

The expected behaviour would be that the package notices that the extension is installed, and that it can do a '/hello' call, so that afterwards the dd-lambda-go package can send traces towards the agent, instead of printing them to stdout for the log forwarder to use.

Actual Behavior

We receive the following debug message: "Will use the API since the Serverless Agent was detected but the hello route was unreachable".

This means that the package correctly detects that the extension is present, but that it doesn't get a 200 OK response from the /hello endpoint of the extension. When we do some debugging ourselves in the lambda, we get a connection refused when trying to query http://localhost:8124/lambda/hello, which explains the debug message.

We were wondering, since @maxday seemed to get it working, if there was something specific for the Golang runtime we need to do, to get this setup working. Based on the documentation of extensions, it should automatically run as a separate process, so not sure what we are doing wrong.

Thanks in advance for any insights!

Steps to Reproduce the Problem

1. Install the Datadog Lambda extension as a layer (arn:aws:lambda:eu-central-1:464622532012:layer:Datadog-Extension:9) => we tried both 8 and 9, same issue.
2. Use the "datadog-lambda-go@maxday/support-tracing-with-extension" package to send traces
3. The package fails to connect to the extension agent

Specifications

• Datadog Go Lambda package version: datadog-lambda-go@maxday/support-tracing-with-extension
• Go version: go1.15.14

Stacktrace

  {
    "date": "2021-07-16T10:14:44Z",
    "os_name": "Linux (Unknown Distribution)",
    "os_version": "2018.03",
    "version": "v1.31.1",
    "lang": "Go",
    "lang_version": "go1.15.14",
    "env": "development",
    "service": "aws.lambda",
    "agent_url": "http://localhost:8126/v0.4/traces",
    "agent_error": "",
    "debug": false,
    "analytics_enabled": false,
    "sample_rate": "NaN",
    "sampling_rules": null,
    "sampling_rules_error": "",
    "tags": {
        "_dd.origin": "lambda",
        "runtime-id": "6474742f-c437-4545-bf05-873c06ba50c7"
    },
    "runtime_metrics_enabled": false,
    "health_metrics_enabled": false,
    "dd_version": "",
    "architecture": "amd64",
    "global_service": "aws.lambda",
    "lambda_mode": "true",
    "agent_features": {
        "DropP0s": false,
        "V05": false,
        "Stats": false
    }
}

[maxday]

Hello @HannesBBR

Thanks for trying our experimental feature. Tracing with go + extension is not yet available and not supported officially, but will be soon :)

According to the behaviour you described, it may come from the runtime you're using in your lambda. I suspect you're using go1.x which is not a supported runtime by AWS. You may want to switch to Custom runtime on Amazon Linux 2 ( provided. al2 ).

[HannesBBR]

Hi Maxime!

Thanks a lot for your quick response! Just after we created the issue we read the same thing in the lambda extension docs. We now see that the agent is running, and get a debug message saying that the package will use the agent.

Unfortunately it seems that our handler function (which we wrap with lambda.Start(ddlambda.WrapHandler(processSQSEvent, nil))) is no longer being called. We're sure WrapHandler is being called because of the debug message Will use the Serverless Agent, but it seems the actual invocation of our own handler is no longer happening.

We of course understand you're not officially supporting this yet, but of course any pointers you could give are highly appreciated (perhaps you ran against the same issue).

EDIT: Apparently we were running an older version of the dd-go-lambda package, updating that solved the issue. Thanks a lot!

[vesquen]

@HannesBBR Are you using the most recent version of "github.com/aws/aws-lambda-go"?