search

open-telemetry / opentelemetry-lambda

Create your own Lambda Layer in each OTel language using this starter code. Add the Lambda Layer to your Lamdba Function to get tracing with OpenTelemetry.
https://opentelemetry.io
Apache License 2.0
283 stars 172 forks source link

Inconsistency in lambda instrumentation across languages. #831

Open biswajit-nanda opened 1 year ago

biswajit-nanda commented 1 year ago

I have three lambda functions exactly with the same business logic written in java, node.js and python instrumented with the latest version of OpenTelemetry lambda auto-instrumentation packages. Each of these lambda functions make two outbound HTTP calls after invocation.

All the three lambda functions are being invoked exactly same way via API Gateway triggers. While they are being instrumented successfully, the spans generated from them are in completely different ways triggering inconsistencies.

Steps to reproduce a. All the three Lambda functions are attached. b. Just auto-instrument them with latest version of opentelemetry auto instrumentation libraries for lambda and configure them to send the traces to an Otel Collector. c. Configure logging exporter in the Otel Collector for trace. e. Configure API Gateway trigger for each of the lambda functions. f. Make a single invocation to each lambda function via the API gateway trigger. g. Check the OpenTelemetry collector logs.

What I expect to See: a. All three lambdas are expected to generate one SERVER type span for the handler triggered by the HTTP gateway and then generate at least two CLIENT type spans for each of the two HTTP outbound calls made. b. All the spans generated should be part of the same TraceID.

What I really see: A. Java Lambda: a. Three SERVER spans are generated from the API gateway trigger, each showing a completely different TraceID. One from instrumentation scope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha and two from instrumentation scope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha. b. Two CLIENT spans for two outbound HTTP calls generated from io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha with the same TraceID as the SERVER Span generated from io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z        info        TracesExporter        {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 3, "spans": 5}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z        debug        otlphttpexporter@v0.77.0/otlp.go:127        Preparing to make HTTP request        {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/metrics"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z        info        ResourceSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Trace ID       : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Parent ID      : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     ID             : 4fbdb5019fb40322
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Name           : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Kind           : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Start time     : 2023-08-04 21:50:13.384374039 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     End time       : 2023-08-04 21:50:13.443981299 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status code    : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:30/ecs/ec2/java/hello)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.peer.port: Int(30)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.response_content_length: Int(76)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Trace ID       : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Parent ID      : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     ID             : 806aede6fce14c90
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Name           : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Kind           : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Start time     : 2023-08-04 21:50:13.48442222 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     End time       : 2023-08-04 21:50:13.503998599 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status code    : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/products.json)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> http.response_content_length: Int(2485)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Trace ID       : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Parent ID      :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     ID             : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Name           : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Kind           : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Start time     : 2023-08-04 21:50:13.365332 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     End time       : 2023-08-04 21:50:13.504745744 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status code    : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Trace ID       : 91228e8b067f5e2d4e92607bda5cd4ba
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Parent ID      :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     ID             : 095c02b9b8f0090d
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Name           : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Kind           : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Start time     : 2023-08-04 21:50:13.365288 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     End time       : 2023-08-04 21:50:13.64412236 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status code    : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:           -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #2
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Trace ID       : 571cc1e464cf9a5a8edaecfc3c25a6b8
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Parent ID      :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     ID             : 0ba08cd04d96c2bf
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Name           : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Kind           : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Start time     : 2023-08-04 21:50:13.365175 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     End time       : 2023-08-04 21:50:13.706600342 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status code    : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:     Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:      -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:           -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]:         {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:20.006Z        debug        otlphttpexporter@v0.77.0/otlp.go:127        Preparing to make HTTP request        {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

B. Node.js Lambda: a. No SERVER type spans are being generated at all. b. Two INTERNAL spans from instrumentation scope @opentelemetry/instrumentation-net 0.32.0 and two CLIENT spans from instrumentation scope @opentelemetry/instrumentation-http 0.41.1 are being generated. The TraceId of one INTERNAL span matches with one of the CLIENT spans and the TraceId of another INTERNAL span matches with the TraceID of the other CLIENT span.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.841Z        info        TracesExporter        {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 4}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z        info        ResourceSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> service.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.language: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.version: Str(1.15.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.provider: Str(aws)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.platform: Str(aws_lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.region: Str(us-west-2)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.version: Str($LATEST)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> service.namespace: Str(aws-lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.pid: Int(8)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.executable.name: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.executable.path: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.command_args: Slice(["/var/lang/bin/node","--expose-gc","--max-http-header-size","81920","--max-semi-space-size=6","--max-old-space-size=116","/var/runtime/index.mjs"])
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.runtime.version: Str(18.16.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.runtime.name: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.runtime.description: Str(Node.js)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.command: Str(/var/runtime/index.mjs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> process.owner: Str(sbx_user1051)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.port: Int(443)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-net 0.32.0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Trace ID       : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Parent ID      : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     ID             : 23382ec398a02d05
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Name           : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Kind           : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Start time     : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     End time       : 2023-08-04 21:42:24.876628992 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status code    : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.host.port: Int(53462)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Trace ID       : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Parent ID      : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     ID             : eea68d81fee6bea3
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Name           : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Kind           : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Start time     : 2023-08-04 21:42:24.536 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     End time       : 2023-08-04 21:42:24.877544192 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status code    : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.port: Str(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.host.port: Int(37510)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-http 0.41.1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Trace ID       : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Parent ID      :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     ID             : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Name           : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Kind           : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Start time     : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     End time       : 2023-08-04 21:42:24.879968768 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status code    : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.target: Str(/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.host: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com:80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.response_content_length_uncompressed: Int(848)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Trace ID       : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Parent ID      :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     ID             : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Name           : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Kind           : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Start time     : 2023-08-04 21:42:24.535000064 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     End time       : 2023-08-04 21:43:06.36994816 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status code    : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.target: Str(/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.host: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.peer.port: Int(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]:         {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z        debug        otlphttpexporter@v0.77.0/otlp.go:127        Preparing to make HTTP request        {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

B. Python Lambda: a. One SERVER type span is generated corresponding to the API Gateway trigger from instrumentation scope opentelemetry.instrumentation.aws_lambda 0.40b0. b. No CLIENT type spans are generated at all. So the two outbound HTTP calls are not being tracked.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z        info        TracesExporter        {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.830Z        info        ResourceSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.language: Str(python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.sdk.version: Str(1.19.0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.region: Str(us-west-2)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.provider: Str(aws)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.version: Str($LATEST)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.instance: Str(2023/08/04/[$LATEST]6e7687d2ff6a4c7e8bc577dacf4625b4)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> service.namespace: Str(aws-lambda)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> service.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> telemetry.auto.version: Str(0.40b0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> appdynamics.controller.port: Int(443)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope opentelemetry.instrumentation.aws_lambda 0.40b0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Trace ID       : e5508d4afeb8280e1370ad838c9dd044
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Parent ID      :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     ID             : 09a7ddff9c33a5b1
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Name           : lambda_function.lambda_handler
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Kind           : Server
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Start time     : 2023-08-04 21:39:06.313833645 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     End time       : 2023-08-04 21:39:06.448371414 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status code    : Unset
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:     Status message :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.execution: Str(a1581204-a256-4c00-937b-a4d6428d4555)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> faas.trigger: Str(http)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.method: Str(GET)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.scheme: Str(https)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> net.host.name: Str(ssn9gz3t4l.execute-api.us-west-2.amazonaws.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.route: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.target: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:      -> http.status_code: Int(200)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]:         {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z        debug        otlphttpexporter@v0.77.0/otlp.go:127        Preparing to make HTTP request        {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

What version of collector/language SDK version did you use? Collector: Latest Version Auto instrumentation library: Latest version Runtime: Java 11, Node.js 18.x, Python 3.11

sourcecode.zip otelcol_log_snippet.zip

github-actions[bot] commented 3 months ago

This issue was marked stale. It will be closed in 30 days without additional activity.

gshpychka commented 3 months ago

Still relevant. Would be nice if issues weren't marked as stale without a comment from a maintainer.

tylerbenson commented 3 months ago

@gshpychka do you have thoughts on how you'd fix this? Open to suggestions.

This particular project doesn't have a lot of backing from big vendors, so the maintainers are mainly here on our free time. As such, we need volunteers to help push things forward.

gshpychka commented 3 months ago

@gshpychka do you have thoughts on how you'd fix this? Open to suggestions.

This particular project doesn't have a lot of backing from big vendors, so the maintainers are mainly here on our free time. As such, we need volunteers to help push things forward.

Oh, please don't take my comment the wrong way - I understand what you wrote, it's just a bit discouraging for issues to be closed without any response. So maybe the stale bot could be tweaked somewhat.

tylerbenson commented 3 months ago

I tried to be very generous on the configuration for the bot. It only marks things as stale when there's been no interaction for a year, and it waits a month before closing something marked as stale. Do you have other suggestions?

gshpychka commented 2 months ago

I tried to be very generous on the configuration for the bot. It only marks things as stale when there's been no interaction for a year, and it waits a month before closing something marked as stale. Do you have other suggestions?

Yes, is it possible to configure it so that it doesn't mark issues as stale unless there's been a comment from a maintainer (or, if that's complicated to define, form any user)?