http.response.status_code vs 'response code' in X-Ray

[ig16022]

Describe the question

Hello, we are attempting to push OpenTelemetry traces from a .NET 8.0 application running in ECS Fargate with AWS Distro for OpenTelemetry running as a sidecar for all services. I am trying to understand the relationship between X-Ray trace information and the metadata that's coming in from our services. As an example, we have "http.response.status_code": 200 in the metadata of a segment, but the X-Ray 'Response Code' in the user interface is always empty. Please see attached screenshot.

We have configured the application to use AWSXRayPropagator, and XRay Trace Id: .AddXRayTraceId() Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

Here is an example of a complete trace displayed by X-Ray:

{ "Id": "1-ee967e4d-1856e32d1b0302ea3f6c599f", "Duration": 0.3, "LimitExceeded": false, "Segments": [ { "Id": "28f47fa7ea0ecebf", "Document": { "id": "28f47fa7ea0ecebf", "name": "Calendar", "start_time": 1705694954.8016534, "trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f", "end_time": 1705694954.8029037, "parent_id": "d5809d2a3518e5b5", "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/515ef0272285473bbd7d7ab5ed56490e", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-calendar-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-calendar-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition" } ] }, "metadata": { "default": { "otel.resource.aws.log.stream.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition:log-stream:ecs/mpa-my_app-calendar-v1_0-container/515ef0272285473bbd7d7ab5ed56490e" ], "otel.resource.service.instance.id": "ip-10-225-0-154", "otel.resource.aws.log.group.names": [ "/ecs/ecs-fargate-my_app-api-calendar-task-definition" ], "otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "otel.resource.aws.ecs.launchtype": "fargate", "url.scheme": "http", "otel.resource.aws.ecs.task.revision": "28", "otel.resource.service.name": "Calendar", "otel.resource.aws.log.group.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition" ], "server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com", "http.request.method": "GET", "otel.resource.cloud.platform": "aws_ecs", "otel.resource.cloud.region": "us-west-1", "http.response.status_code": 200, "otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/515ef0272285473bbd7d7ab5ed56490e", "otel.resource.telemetry.sdk.name": "opentelemetry", "otel.resource.service.version": "1.0.0.0", "otel.resource.cloud.availability_zone": "us-west-1a", "network.protocol.version": "1.1", "otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-calendar-task-definition", "otel.resource.cloud.account.id": "MY_ACCOUNT_ID", "otel.resource.telemetry.sdk.version": "1.7.0", "url.path": "/calendar/index.html", "otel.resource.telemetry.sdk.language": "dotnet", "otel.resource.aws.log.stream.names": [ "ecs/mpa-my_app-calendar-v1_0-container/515ef0272285473bbd7d7ab5ed56490e" ], "otel.resource.cloud.provider": "aws" } }, "service": { "version": "1.0.0.0" }, "origin": "AWS::ECS::Fargate" } }, { "Id": "9266d561577b88d1", "Document": { "id": "9266d561577b88d1", "name": "Dashboard", "start_time": 1705694954.6465578, "trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f", "end_time": 1705694954.9463103, "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-dashboard-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" } ] }, "metadata": { "default": { "otel.resource.aws.log.stream.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition:log-stream:ecs/mpa-my_app-dashboard-v1_0-container/a51a6e88ad5341a4b2374774571096be" ], "otel.resource.host.name": "ip-10-225-0-238.us-west-1.compute.internal", "otel.resource.service.instance.id": "ip-10-225-0-238", "otel.resource.aws.log.group.names": [ "/ecs/ecs-fargate-my_app-api-dashboard-task-definition" ], "otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "otel.resource.aws.ecs.launchtype": "fargate", "url.scheme": "http", "otel.resource.aws.ecs.task.revision": "36", "otel.resource.service.name": "Dashboard", "otel.resource.aws.log.group.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" ], "server.address": "l04v5pn392.execute-api.us-west-1.amazonaws.com", "url.query": "?userId=22", "http.request.method": "GET", "otel.resource.cloud.platform": "aws_ecs", "otel.resource.os.type": "linux", "otel.resource.cloud.region": "us-west-1", "http.response.status_code": 200, "otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "otel.resource.telemetry.sdk.name": "opentelemetry", "user_agent.original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", "otel.resource.service.version": "1.0.0.0", "otel.resource.cloud.availability_zone": "us-west-1a", "network.protocol.version": "1.1", "otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-dashboard-task-definition", "otel.resource.cloud.account.id": "MY_ACCOUNT_ID", "otel.resource.telemetry.sdk.version": "1.7.0", "url.path": "/dashboard/api/v1.0/dashboard", "http.route": "api/v{version:apiVersion}/dashboard", "otel.resource.telemetry.sdk.language": "dotnet", "otel.resource.aws.log.stream.names": [ "ecs/mpa-my_app-dashboard-v1_0-container/a51a6e88ad5341a4b2374774571096be" ], "otel.resource.cloud.provider": "aws" } }, "service": { "version": "1.0.0.0" }, "origin": "AWS::ECS::Fargate", "subsegments": [ { "id": "fe055153f19488ea", "name": "Get Online Application Data", "start_time": 1705694954.8839161, "end_time": 1705694954.8855886, "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-dashboard-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" } ] } },
{ "id": "3c7c5884e1f95e50", "name": "GET", "start_time": 1705694954.7782323, "end_time": 1705694954.798333, "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-dashboard-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" } ] }, "metadata": { "default": { "server.address": "internal-my_app-dev-alb1-123499307.us-west-1.elb.amazonaws.com", "http.request.method": "GET", "network.protocol.version": "1.1", "http.response.status_code": 301, "url.full": "http://internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com/calendar/?userId=22" } }, "namespace": "remote" }, { "id": "cfd82c6836bc3bc1", "name": "Get Navigatables Data", "start_time": 1705694954.8856056, "end_time": 1705694954.8867292, "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-dashboard-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" } ] } }, { "id": "d5809d2a3518e5b5", "name": "GET", "start_time": 1705694954.8006032, "end_time": 1705694954.8034613, "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be", "availability_zone": "us-west-1a", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-dashboard-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition" } ] }, "metadata": { "default": { "server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com", "http.request.method": "GET", "network.protocol.version": "1.1", "http.response.status_code": 200, "url.full": "http://internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com/calendar/index.html" } }, "namespace": "remote" } ] } }, { "Id": "d8281f3d5cf86a7f", "Document": { "id": "d8281f3d5cf86a7f", "name": "Calendar", "start_time": 1705694954.7947981, "trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f", "end_time": 1705694954.796209, "parent_id": "3c7c5884e1f95e50", "fault": false, "error": false, "throttle": false, "aws": { "account_id": "MY_ACCOUNT_ID", "ecs": { "task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/2cadcc3b1ab1465298d69a5c9675ce34", "availability_zone": "us-west-1c", "cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "task_family": "ecs-fargate-my_app-api-calendar-task-definition", "launch_type": "fargate" }, "xray": { "auto_instrumentation": false, "sdk_version": "1.7.0", "sdk": "opentelemetry for dotnet" }, "cloudwatch_logs": [ { "log_group": "/ecs/ecs-fargate-my_app-api-calendar-task-definition", "arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition" } ] }, "metadata": { "default": { "otel.resource.aws.log.stream.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition:log-stream:ecs/mpa-my_app-calendar-v1_0-container/2cadcc3b1ab1465298d69a5c9675ce34" ], "otel.resource.service.instance.id": "ip-10-225-2-60", "otel.resource.aws.log.group.names": [ "/ecs/ecs-fargate-my_app-api-calendar-task-definition" ], "otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster", "otel.resource.aws.ecs.launchtype": "fargate", "url.scheme": "http", "otel.resource.aws.ecs.task.revision": "28", "otel.resource.service.name": "Calendar", "otel.resource.aws.log.group.arns": [ "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition" ], "server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com", "url.query": "?userId=22", "http.request.method": "GET", "otel.resource.cloud.platform": "aws_ecs", "otel.resource.cloud.region": "us-west-1", "http.response.status_code": 301, "otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/2cadcc3b1ab1465298d69a5c9675ce34", "otel.resource.telemetry.sdk.name": "opentelemetry", "otel.resource.service.version": "1.0.0.0", "otel.resource.cloud.availability_zone": "us-west-1c", "network.protocol.version": "1.1", "otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-calendar-task-definition", "otel.resource.cloud.account.id": "MY_ACCOUNT_ID", "otel.resource.telemetry.sdk.version": "1.7.0", "url.path": "/calendar/", "otel.resource.telemetry.sdk.language": "dotnet", "otel.resource.aws.log.stream.names": [ "ecs/mpa-my_app-calendar-v1_0-container/2cadcc3b1ab1465298d69a5c9675ce34" ], "otel.resource.cloud.provider": "aws" } }, "service": { "version": "1.0.0.0" }, "origin": "AWS::ECS::Fargate" } } ] }

What did you expect to see? Expecting to see Response Code, Request URl, Request method, etc. populated, but these are always emtpy in X-Ray user interface. These values can be seen in the metadata only.

Environment Describe any aspect of your environment. If this is related to a deployment of the ADOT Collector please provide your Collector config file.

I have tried several variations of the predefined collector config files, but the results are exactly the same.

"--config=/etc/ecs/ecs-default-config.yaml" --config=/etc/ecs/ecs-xray.yaml "--config=/etc/ecs/container-insights/otel-task-metrics-config.yaml"

[rafar55]

@ig16022 we are having exactly the same issue as you. We're you able to find a solution to this?

[zeva-us]

I am facing the same issue. It looks like the format expected by Xray is like this (directly in the segment document) :

"http": { "request": { "url": "http://certificates.godaddy.com/repository/gdig2.crt", "method": "GET" }, "response": { "status": 200, "content_length": 0 } },

Meanwhile the collector is producing the http information in the metadata: "http.request.method": "POST", "http.route": "GetCertificatesSummary", "http.response.status_code": 200,

Let me know if this is a problem in our configuration or this is something that should be fixed in the collector itself.

[julianh-y]

I'm having the same issue.

According to Semantic Conventions for HTTP Spans the key-values are correct, so maybe there is an issue with the exporter to X-Ray?. X-Ray seems to want this information as nested objects...

[zeva-us]

I think the issue is in the collector, since it translates the Otel attributes ("http.request.method": "POST" for example) to metadata attributes and not in a form the Xray Portal understands it.

[jorgeparavicini]

I am having the same problem. Our .net OTel produces the attributes in the form {"http.request.method": "POST"} but according to the aws-otel-collector docs it expects the attribute to be http.method instead. So the issue seems to be in the collector

[zeva-us]

I am upvoting this issue again.

[kcrandall]

I'm having the same issue with this. ASP.netCore otel outputs with http.request.method and no meta data is showing up in xray (method, url, etc)

[ig16022]

aws-otel-collector team - Are you able to provide any feedback?

[jorgeparavicini]

Any news on this? I believe that the actual implementation of the collector is located here maybe we have to check there

[Elter71]

Here is commit that should fix this issue: https://github.com/open-telemetry/opentelemetry-collector-contrib/commit/fe357169072893b908e855c65e87070b69685c34

Release: https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.96.0

[zeva-us]

Thank you Piotr! thats good new, but we should update aws-otel-collector to use the new fixed opentelemetry collector. is that correct?

[Elter71]

Thank you Piotr! thats good new, but we should update aws-otel-collector to use the new fixed opentelemetry collector. is that correct?

Unfortunately, we have to wait until aws-otel-collector releases a new version with that fix.

[EduardoMRB]

Also having the same problem. Hopefully that gets released soon :pray:

[Elter71]

Hi, all! The OTel Collector has finally been updated to the new version v0.98.0 in the latest release v0.39.0. Everything should be working now.