Fail on send telemetry on "warm start" and "shutdown" after timeout

[Vladikamira]

The context: datadog-lambda-extension version 44 Lambda and DD site is located in the EU.

the problem: It looks like dd-extension starts telemetry flush and does not stop before each invocation ends when Lambda enters the IDLE state. We see that by Warnings and Errors messages when the function gets invoked either after the warm start on subsequent invocation or the shutdown event. Error/Warning pops up when more than WaitTimeout (5 seconds) has passed after the previous invocation (I didn't dive deep into the code, but that looks like the proper threshold).

Example of the WARN:

DD_EXTENSION | WARN | Could not send payload: Post "https://http-intake.logs.datadoghq.eu/api/v2/logs": context deadline exceeded (Client. Timeout exceeded while awaiting headers)

example of the ERROR:

 | UTC | DD_EXTENSION | ERROR | Exporting failed. No more retries left. Dropping data.

Here are some examples in screenshots:

WARN on subsequent invocation:

WARN on shutdown event:

ERROR on shutdown (this one seems to be from the OTLP part of the agent):

[Vladikamira]

problem reproduced on the datadog-lambda-extension version 53 (cannot check 55 as the image is not published)

[zARODz11z]

Hi @Vladikamira i'm able to find v55's tag here https://hub.docker.com/r/datadog/lambda-extension/tags. May you clarify if we are on the same page about

cannot check 55 as the image is not published

[Vladikamira]

Hi @Vladikamira i'm able to find v55's tag here https://hub.docker.com/r/datadog/lambda-extension/tags. May you clarify if we are on the same page about

cannot check 55 as the image is not published

indeed, I will try that one from DockerHub, thanks! 👍 We are using the AWS one: https://gallery.ecr.aws/datadog/lambda-extension, the last one there is 53

[Vladikamira]

version 55 has the same problem

[hghotra]

Hi @Vladikamira, can you try the latest version of the extension and report back? This was fixed in a recent version of the extension.

[DylanLovesCoffee]

Upgrading to v59+ should include a change that improves the flushing logic for sending logs. Could we give that a try and then keep us updated?

[Vladikamira]

Thanks! yup, I cannot reproduce the issue anymore on the version v64 🎉

but I got new WARN messages though 😅

2024-09-11 15:09:38 UTC | DD_EXTENSION | WARN | config key flare_stripped_keys is unknown
2024-09-11 15:09:38 UTC | DD_EXTENSION | WARN | failed to get configuration value for key "flare_stripped_keys": unable to cast <nil> of type <nil> to []string
2024-09-11 15:09:38 UTC | DD_EXTENSION | WARN | config key scrubber.additional_keys is unknown
2024-09-11 15:09:38 UTC | DD_EXTENSION | WARN | failed to get configuration value for key "scrubber.additional_keys": unable to cast <nil> of type <nil> to []string

[Vladikamira]

anyway, this issue is fixed, therefore I'm closing PR, thanks! 🙏 I'Il open another one for a new issue 😄

[Vladikamira]

sorry, we do not have these massages anymore after the upgrade to v64

DD_EXTENSION | WARN | Could not send payload: Post "https://http-intake.logs.datadoghq.eu/api/v2/logs": EOF (Client.Timeout exceeded while awaiting headers)

but we still have these

 DD_EXTENSION | WARN | SyncForwarder.sendHTTPTransactions failed to send: error while sending transaction, rescheduling it: Post "https://7-55-3-app.agent.datadoghq.eu/api/v1/series": context deadline exceeded (Client.Timeout exceeded while awaiting headers)