New Observability Features in AWS SDK for .NET

[muhammad-othman]

Observability Features in AWS SDK for .NET

We are excited to announce the release of the AWS SDK for .NET version 3.7.400, which contains significant observability enhancements. Our goal is to streamline the integration with OpenTelemetry, providing users with powerful tools to monitor, trace, and measure their applications' performance effectively. This initiative aims to offer a seamless experience, ensuring that users can gain deep insights into their applications with minimal configuration.

High Level Overview

• Simplified Setup: Users will be able to enable tracing and metrics collection with minimal effort by adding simple extension methods in their configuration. For example:

  
  var openTelemetryBuilder = builder.Services.AddOpenTelemetry()
  .ConfigureResource(resource => resource.AddService(serviceName));

openTelemetryBuilder.WithTracing(builder => builder .AddAWSInstrumentation() .AddAspNetCoreInstrumentation() .AddConsoleExporter());

openTelemetryBuilder.WithMetrics(builder => builder .AddAWSInstrumentation() .AddAspNetCoreInstrumentation() .AddConsoleExporter());

* Dependency Requirement: Users will need to add a dependency for AWS instrumentation. The specific dependency is still to be determined.
* Flexible Integration: Users will be able to direct traces and metrics to any OpenTelemetry-supported exporter. This ensures compatibility with a wide range of observability tools and platforms.
* Metrics and Tracing: Users will benefit from both tracing and metrics, enabling them to monitor performance, detect anomalies, and diagnose issues effectively.
    * Spans:
        * Service Operations.
        *  Credentials retrieval.
        *  Compression.
        *  Http Requests.
    * Metrics:
         * client.call.duration
        * client.uptime
        * client.call.attempts
        * client.call.errors
        * client.call.attempt_duration
        * client.call.resolve_endpoint_duration
        * client.call.serialization_duration
        * client.call.deserialization_duration
        * client.call.auth.signing_duration
        * client.call.auth.resolve_identity_duration
        * client.http.bytes_sent
        * client.http.bytes_received

We are currently working on the initial implementation and will share more detailed updates as we progress. In the meantime, we welcome your feedback and suggestions on how we can make this integration as beneficial and user-friendly as possible.

For more details and to track our progress, please check out [the feature branch](https://github.com/aws/aws-sdk-net/tree/DOTNET-7462-Observability-for-the-NET-SDK) 
And the merged PRs: 
* https://github.com/aws/aws-sdk-net/pull/3352 
* https://github.com/aws/aws-sdk-net/pull/3344
* https://github.com/aws/aws-sdk-net/pull/3379
* https://github.com/aws/aws-sdk-net/pull/3391

[michaeldimoudis]

Apologies if this is not the right place to ask, or if I’m too soon and this info is coming. I tried integrating the 3.7.400 version nuget packages with OpenTelemetry, but I have no idea how to hook this up. I tried TelemetryProvider however the TracerProvider between AWS and OpenTelemetry is different and couldn’t get it to work. Will there be any information coming how to integrate this with OpenTelemetry?

[muhammad-othman]

Apologies if this is not the right place to ask, or if I’m too soon and this info is coming. I tried integrating the 3.7.400 version nuget packages with OpenTelemetry, but I have no idea how to hook this up. I tried TelemetryProvider however the TracerProvider between AWS and OpenTelemetry is different and couldn’t get it to work. Will there be any information coming how to integrate this with OpenTelemetry?

Hi @michaeldimoudis, Thanks for reaching out!

We are actively working on extending OpenTelemetry.Instrumentation.AWS package to support the new tracing and metrics capabilities introduced in AWS SDK version 3.7.400. Several PRs to add these new changes are currently in progress.

In the meantime, you can use the current version of OpenTelemetry.Instrumentation.AWS package for integrating AWS services with OpenTelemetry. This package currently supports instrumentation for service operation spans and does not yet include metrics instrumentation.

The upcoming changes will not alter the AddAWSInstrumentation extension method used for integrating this package into your application. This means you can start using it now for limited tracing coverage, and when you upgrade to the new version that includes our PRs, you will receive the new spans without any changes to your code.

If you need any help with the integration or have any questions, feel free to ask!

[michaeldimoudis]

Thanks for the quick response @muhammad-othman!

Sounds great! I'm already using OpenTelemetry.Instrumentation.AWS and I didn't see a difference in traces between the old and 3.7.400 hence why I asked :) I'm looking forward to the future updates.

And just for curiosity, will the work in extending that package include Semantic Conventions for Messaging Spans as listed here https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#messaging-attributes such as messaging.system, messaging.destination.name, messaging.batch.message_count etc?

[muhammad-othman]

Thank you @michaeldimoudis for your feedback!

The upcoming changes focus on attributes that apply universally to all services. While these updates won't include service-specific attributes, we are considering adding such attributes, including those for messaging services, in future releases.

[muhammad-othman]

Hi @michaeldimoudis, The new Observability changes was released as part of version 1.1.0-beta.5 of OpenTelemetry.Instrumentation.AWS It includes the new traces in addition to the ability to add AWS instrumentation for metrics too.