search

aws / copilot-cli

The AWS Copilot CLI is a tool for developers to build, release and operate production ready containerized applications on AWS App Runner or Amazon ECS on AWS Fargate.
https://aws.github.io/copilot-cli/
Apache License 2.0
3.48k stars 401 forks source link

[Bug]: The version of AWS ADOT used for observability is incompatible with the latest version of dotnet opentelemetry library. #5905

Open ksmith97 opened 1 month ago

ksmith97 commented 1 month ago

Description:

When deploying a dotnet 8.0 core ASP application using OpenTelemetry.Exporter.OpenTelemtryProtocol 1.9.0 (The latest version as of this ticket) with observability enabled, traces will not be transferred properly into AWS Xray. Overriding the image to deploy the latest version v0.40.0(as of this ticket) resolved the issue.

Details:

There are no errors that are made by the library or by the collector but even after tweaking a number of settings(including trying grpc vs protobuf) it does not seem to work. The version of the ADOT image is v.0.14 and from 2021 and looks to be quite a bit out of date.
By comparison, the example nodejs application in the doc does work properly when deployed but if you deploy a simple dotnet application (such as the example app at https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/getting-started-aspnetcore) almost nothing makes it into xray(Every so often a empty message with the correct name would make it through. No idea how or why)

Observed result:

No traces or metrics in AWS Xray and no error messages.

Expected result:

Traces and metrics in AWS Xray

Debugging:

Enabled logging in the OLTP library in the dotnet application and saw no errors. There are also no errors in the collector's logs. Created service override to deploy latest version of ADOT resolved issue.

ksmith97 commented 1 month ago

For anyone else that experiences this I resolved the issue in my project by adding the override

- op: replace
  path: /Resources/TaskDefinition/Properties/ContainerDefinitions/1/Image
  value: public.ecr.aws/aws-observability/aws-otel-collector:v0.40.0

This may need to have the index adjusted based on your config.