Utility for building a lambda layer artifact

srprash commented 1 week ago

Description of changes

Adding a script and utilities to build a .zip artifact that will be published as the ADOT Java Lambda Layer.
The layer packages patched version of upstream (opentelemetry-java-instrumentation) and the current source build of aws-opentelemetry-java-instrumentation, including the Application Signals logic.
The layer building logic is primarily borrowed from what we have in the aws-otel-lambda repo, and is tweaked to the build a javaagent instrumentation only layer from the current repo.

Testing

Setup

run lambda-layer/build-layer.sh
- This will produce a opentelemetry-javaagent-layer.zip artifact.
From the AWS Lambda console, created a Java 17 function. The code makes a remote call using OkHttp to www.amazon.com
From the AWS Lambda console, create a lambda layer using the opentelemetry-javaagent-layer.zip file obtained in previous step.

Set the following environment variables on the lambda function:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true
OTEL_LOGS_EXPORTER=none
OTEL_METRICS_EXPORTER=none
OTEL_TRACES_EXPORTER=otlp

Enable active tracing on the function
Invoke the function from lambda console

Result

Trace view

Trace timeline view

Trace raw data

{
    "Id": "1-673ccffc-3f2d31933a2b0f341ee61980",
    "Duration": 7.728,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "8aba0f96e08d63a1",
            "Document": {
                "id": "8aba0f96e08d63a1",
                "name": "java-lambda-example",
                "start_time": 1732038657.532657,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.474236,
                "parent_id": "31aa4a42384b852d",
                "aws": {
                    "cloudwatch_logs": [
                        {
                            "log_group": "/aws/lambda/java-lambda-example"
                        }
                    ]
                },
                "annotations": {
                    "aws:responseLatency": 2939.22,
                    "aws:runtimeOverhead": 1.565,
                    "aws:responseDuration": 0.101
                },
                "origin": "AWS::Lambda::Function",
                "subsegments": [
                    {
                        "id": "55219d8790c63176",
                        "name": "Init",
                        "start_time": 1732038653.075805,
                        "end_time": 1732038657.52772
                    },
                    {
                        "id": "8397e02d88907ef8",
                        "name": "Overhead",
                        "start_time": 1732038660.47,
                        "end_time": 1732038660.471565
                    }
                ]
            }
        },
        {
            "Id": "31aa4a42384b852d",
            "Document": {
                "id": "31aa4a42384b852d",
                "name": "java-lambda-example",
                "start_time": 1732038652.748,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.476,
                "http": {
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "request_id": "e37debae-de16-47fb-8a15-be426c90df82"
                },
                "origin": "AWS::Lambda",
                "resource_arn": "arn:aws:lambda:us-west-2:702258172533:function:java-lambda-example"
            }
        },
        {
            "Id": "0a87c356b3cbd3eb",
            "Document": {
                "id": "0a87c356b3cbd3eb",
                "name": "java-lambda-example",
                "start_time": 1732038657.649404,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.007994,
                "parent_id": "8aba0f96e08d63a1",
                "aws": {
                    "span.kind": "LOCAL_ROOT"
                },
                "annotations": {
                    "aws.local.service": "java-lambda-example",
                    "span.name": "java-lambda-example",
                    "aws.local.operation": "java-lambda-example/FunctionHandler",
                    "span.kind": "SERVER",
                    "aws.local.environment": "lambda:default"
                },
                "metadata": {
                    "process.command_args": [
                        "/var/lang/bin/java",
                        "-XX:MaxHeapSize=445645k",
                        "-javaagent:/var/runtime/lib/Log4jHotPatch.jar=log4jFixerVerbose=false",
                        "-XX:+UseSerialGC",
                        "-Xshare:on",
                        "-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa",
                        "-XX:+TieredCompilation",
                        "-XX:TieredStopAtLevel=1",
                        "--add-opens=java.base/java.io=ALL-UNNAMED",
                        "-Dorg.crac.Core.Compat=com.amazonaws.services.lambda.crac",
                        "-XX:+ErrorFileToStderr",
                        "-Dcom.amazonaws.services.lambda.runtime.api.client.runtimeapi.NativeClient.JNI=/var/runtime/lib/jni/libaws-lambda-jni.linux-x86_64.so",
                        "-classpath",
                        "/var/runtime/lib/aws-lambda-java-core-1.2.3.jar:/var/runtime/lib/aws-lambda-java-runtime-interface-client-2.6.0-linux-x86_64.jar:/var/runtime/lib/aws-lambda-java-serialization-1.1.5.jar",
                        "com.amazonaws.services.lambda.runtime.api.client.AWSLambda",
                        "com.example.LambdaHandler"
                    ],
                    "process.runtime.version": "17.0.13+11-LTS",
                    "os.type": "linux",
                    "thread.name": "main",
                    "process.pid": 2,
                    "telemetry.sdk.name": "opentelemetry",
                    "telemetry.sdk.language": "java",
                    "process.runtime.name": "OpenJDK Runtime Environment",
                    "os.description": "Linux 5.10.227-239.884.amzn2.x86_64",
                    "host.arch": "amd64",
                    "cloud.resource_id": "arn:aws:lambda:us-west-2:702258172533:function:java-lambda-example",
                    "host.name": "169.254.30.77",
                    "telemetry.sdk.version": "1.34.1",
                    "cloud.platform": "aws_lambda",
                    "telemetry.auto.version": "1.32.1-adot-lambda1-aws",
                    "thread.id": 1,
                    "cloud.region": "us-west-2",
                    "service.name": "java-lambda-example",
                    "faas.name": "java-lambda-example",
                    "cloud.provider": "aws",
                    "faas.invocation_id": "e37debae-de16-47fb-8a15-be426c90df82",
                    "cloud.account.id": "702258172533",
                    "process.executable.path": "/var/lang/bin/java",
                    "faas.version": "$LATEST",
                    "PlatformType": "AWS::Lambda",
                    "process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.13+11-LTS"
                },
                "subsegments": [
                    {
                        "id": "5891a91ff13a3361",
                        "name": "www.amazon.com:443",
                        "start_time": 1732038659.8066413,
                        "end_time": 1732038659.972645,
                        "http": {
                            "request": {
                                "url": "https://www.amazon.com/",
                                "method": "GET"
                            },
                            "response": {
                                "status": 200
                            }
                        },
                        "aws": {
                            "span.kind": "CLIENT"
                        },
                        "annotations": {
                            "aws.local.service": "java-lambda-example",
                            "span.name": "GET",
                            "aws.local.operation": "java-lambda-example/FunctionHandler",
                            "span.kind": "CLIENT",
                            "aws.remote.service": "www.amazon.com:443",
                            "aws.remote.operation": "GET /",
                            "aws.local.environment": "lambda:default"
                        },
                        "metadata": {
                            "http.url": "https://www.amazon.com/",
                            "process.command_args": [
                                "/var/lang/bin/java",
                                "-XX:MaxHeapSize=445645k",
                                "-javaagent:/var/runtime/lib/Log4jHotPatch.jar=log4jFixerVerbose=false",
                                "-XX:+UseSerialGC",
                                "-Xshare:on",
                                "-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa",
                                "-XX:+TieredCompilation",
                                "-XX:TieredStopAtLevel=1",
                                "--add-opens=java.base/java.io=ALL-UNNAMED",
                                "-Dorg.crac.Core.Compat=com.amazonaws.services.lambda.crac",
                                "-XX:+ErrorFileToStderr",
                                "-Dcom.amazonaws.services.lambda.runtime.api.client.runtimeapi.NativeClient.JNI=/var/runtime/lib/jni/libaws-lambda-jni.linux-x86_64.so",
                                "-classpath",
                                "/var/runtime/lib/aws-lambda-java-core-1.2.3.jar:/var/runtime/lib/aws-lambda-java-runtime-interface-client-2.6.0-linux-x86_64.jar:/var/runtime/lib/aws-lambda-java-serialization-1.1.5.jar",
                                "com.amazonaws.services.lambda.runtime.api.client.AWSLambda",
                                "com.example.LambdaHandler"
                            ],
                            "process.runtime.version": "17.0.13+11-LTS",
                            "os.type": "linux",
                            "thread.name": "main",
                            "process.pid": 2,
                            "telemetry.sdk.name": "opentelemetry",
                            "net.peer.name": "www.amazon.com",
                            "telemetry.sdk.language": "java",
                            "process.runtime.name": "OpenJDK Runtime Environment",
                            "os.description": "Linux 5.10.227-239.884.amzn2.x86_64",
                            "host.arch": "amd64",
                            "net.protocol.name": "http",
                            "host.name": "169.254.30.77",
                            "telemetry.sdk.version": "1.34.1",
                            "http.method": "GET",
                            "cloud.platform": "aws_lambda",
                            "telemetry.auto.version": "1.32.1-adot-lambda1-aws",
                            "thread.id": 1,
                            "net.peer.port": 443,
                            "cloud.region": "us-west-2",
                            "service.name": "java-lambda-example",
                            "faas.name": "java-lambda-example",
                            "http.status_code": 200,
                            "cloud.provider": "aws",
                            "process.executable.path": "/var/lang/bin/java",
                            "faas.version": "$LATEST",
                            "PlatformType": "AWS::Lambda",
                            "process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.13+11-LTS",
                            "net.protocol.version": "2"
                        },
                        "namespace": "remote"
                    }
                ]
            }
        },
        {
            "Id": "2f12e028332d91f3",
            "Document": {
                "id": "2f12e028332d91f3",
                "name": "www.amazon.com:443",
                "start_time": 1732038659.8066413,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038659.972645,
                "parent_id": "5891a91ff13a3361",
                "inferred": true,
                "http": {
                    "request": {
                        "url": "https://www.amazon.com/",
                        "method": "GET"
                    },
                    "response": {
                        "status": 200
                    }
                },
                "annotations": {
                    "aws.local.service": "www.amazon.com:443",
                    "aws.local.operation": "GET /"
                }
            }
        }
    ]
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

bjrara commented 1 week ago

Should we also set OTEL_AWS_APPLICATION_SIGNALS_RUNTIME_ENABLED to false on Lambda?

srprash commented 1 week ago

Should we also set OTEL_AWS_APPLICATION_SIGNALS_RUNTIME_ENABLED to false on Lambda?

Ah yes. That's why my lambda logs were noisy.

codecov-commenter commented 1 week ago

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 69.66%. Comparing base (09e6487) to head (d662fe9). Report is 371 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #953 +/- ## ============================================= - Coverage 85.71% 69.66% -16.05% - Complexity 19 308 +289 ============================================= Files 3 26 +23 Lines 49 1134 +1085 Branches 5 149 +144 ============================================= + Hits 42 790 +748 - Misses 3 294 +291 - Partials 4 50 +46 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

aws-observability / aws-otel-java-instrumentation