search

DataDog / datadog-serverless-functions

Repo of AWS Lambda and Azure Functions functions that process streams and send data to Datadog
Apache License 2.0
338 stars 387 forks source link

AWS DataDog Forwarder failing due to old URLlib3 #847

Open rknechtel opened 1 month ago

rknechtel commented 1 month ago

Describe what happened: Using latest release of AWS DataDog Forwarder Lambda code, all lambda calls are failing due to the use of an old urllib3 version.

Log Group: /aws/lambda/DatadogIntegration-ForwarderStack-H16PMQ-Forwarder-PYitVeIgHSnt

Errors in my CloudWatch logs

START RequestId: 21fd84d1-1a80-44c3-9ca6-67d64f4f65c4 Version: $LATEST
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips  26 Jan 2017'. See: https://github.com/urllib3/urllib3/issues/2168
Traceback (most recent call last):
END RequestId: 21fd84d1-1a80-44c3-9ca6-67d64f4f65c4

When trying to create Lambda code from source when I run pip3 install -r requirements.txt I see this error: ERROR: opentelemetry-api 1.27.0 has requirement importlib-metadata<=8.4.0,>=6.0, but you'll have importlib-metadata 8.5.0 which is incompatible. ERROR: datadog-lambda 5.87.0 has requirement urllib3<2.0.0; python_version < "3.11", but you'll have urllib3 2.2.3 which is incompatible.

The requirements.txt has these: datadog-lambda==5.87.0 urllib3>=1.26.19,<3.0

The urlib3 allows versions up to 3.0 but the datadog-lambda version only allows urllib versions up to 2.0 - which is where the CloudWatch log errors are coming from (urllib3 2.0).

I made updates to the requirements.txt locally and then built the code. I made the following changes: Latest version of datadog-lambda: 6.98.0

Original in requirements.txt: datadog-lambda==5.87.0 ddtrace==2.6.5

Updated: datadog-lambda==6.98.0 ddtrace>=2.10.0

It will build then. But when uploading that code to the Lambda I still get the errors.

Describe what you expected: Lambda should work with a new urllib3 version.

Steps to reproduce the issue: deploy latest release code (aws-dd-forwarder-3.123.0.zip) to DataDog AWS lambda (DatadogIntegration-ForwarderStack-H16PMQ-Forwarder-PYitVeIgHSnt).

Look in CloudWatch logs for error.

rknechtel commented 1 month ago

I got it working here is what I did: Updated to latest CloudFormation Template: https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml

git cloned the repo. Updated the requirements.txt: Original in requirements.txt: datadog-lambda==5.87.0 ddtrace==2.6.5

Updated: datadog-lambda==6.98.0 ddtrace>=2.10.0

Built lambda code: python3.11 -m venv ~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env source ~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/bin/activate ~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/bin/pip3.11 install -r requirements.txt

Copied: init.py customized_log_group.py enhanced_lambda_metrics.py forwarder.py lambda_function.py requirements.txt settings.py setup.py telemetry.py template.yaml

To: /projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/lib/python3.11/site-packages

Then zipped up everything in that directory and deployed it to the lambda. Now my CloudWatch Logs are getting to DataDog Log Explorer.