awslabs / aws-lambda-rust-runtime

A Rust runtime for AWS Lambda
Apache License 2.0
3.29k stars 335 forks source link

Add OpenTelemetryLayer to lambda_runtime layers #866

Closed ollis closed 1 month ago

ollis commented 3 months ago

Hi, I have started testing OpenTelemetry integration and I found the example given here really useful: https://github.com/awslabs/aws-lambda-rust-runtime/blob/main/examples/opentelemetry-tracing/src/lib.rs

I have copied the code into my project but I am wondering if it could be packaged into the distributed crate, maybe behind an opentelemetry feature flag?

I have also tried using nested layers and I would expect this to create child spans but this is not what I am observing. Is this a feature or a bug?

Given this setup I would expect the span of Tracing-layer to be a child of the OpenTelemetry-layer span:

 let runtime = Runtime::new(handler)
            .layer(layers::TracingLayer::new())
            .layer(OpenTelemetryLayer::new(|| {
                tracer
                    .provider()
                    .expect("Unable to get tracer provider!")
                    .force_flush();
            }));

But this is the result (function_name is customer-lookup):

Screenshot 2024-05-02 at 08 11 20
calavera commented 3 months ago

Providing that layer can be a good idea since multiple people seem to use it.

OpenTelemetry and Tracing are two completely separated tracing systems. Adding both layers is not going to do what you expect. You need to use the Tracing / Open Telemetry integration for that: https://docs.rs/tracing-opentelemetry/latest/tracing_opentelemetry/

ollis commented 3 months ago

Both TracingLayer and OpenTelemetryLayer uses tracing::info_span! from the tracing crate. So if I created MyCustomLayer and added some tracing::info_span! statements I would expect them to be nested within spans from an outer layer? (I used TracingLayer in the example just for simplicity) I have already set up tracing_opentelemetry in order to emit the events to Jaeger where the attached image is taken.

github-actions[bot] commented 1 month ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one.