aws-observability / aws-otel-js

AWS Distro for OpenTelemetry JavaScript SDK
https://aws-otel.github.io/
Apache License 2.0
36 stars 22 forks source link

Http status code is always 0 in X-ray for JS otel instrumentation #52

Closed niekname closed 3 years ago

niekname commented 3 years ago

I have a node.js api (express) running on ecs fargate. I am running the aws otel collector (v.0.7.0) as sidecar container and have setup the javascript tracing as explained in the documentation. Everything works fine, but the http status code always seems to be 0 in x-ray, although it is included in the otel instrumentation. When I use the console exporter you can see that the http status code is there, so it gets lost somewhere when sending it to x-ray, or x-ray doesn't process it correctly, not sure where to look.

example of console export: { traceId: '6675317b0ad8205b6851b809395be4b7', parentId: undefined, name: 'HTTP GET', id: '9aed987b4122245c', kind: 1, timestamp: 1614809120176327, duration: 6635, attributes: { 'http.url': 'http://localhost:8080/woops', 'http.host': 'localhost:8080', 'net.host.name': 'localhost', 'http.method': 'GET', 'http.route': '', 'http.target': '/woops', 'http.user_agent': 'curl/7.64.1', 'http.flavor': '1.1', 'net.transport': 'IP.TCP', 'net.host.ip': '::ffff:127.0.0.1', 'net.host.port': 8080, 'net.peer.ip': '::ffff:127.0.0.1', 'net.peer.port': 56774, 'http.status_code': 401, 'http.status_text': 'UNAUTHORIZED' }, status: { code: 2 }, events: [] }

I am using the http, https, express auto instrumentation plugins.

There is a printscreen of a trace in the documentation which shows the same problem. The response code is also 0. https://aws-otel.github.io/docs/getting-started/js-sdk/trace-manual-instr#start-tracing

Excerpt of raw data of x-ray trace: "http": { "request": { "url": "http://localhost:8080/woops", "method": "GET", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36", "client_ip": "127.0.0.1", "x_forwarded_for": true }, "response": { "status": 0, "content_length": 0 } }, "aws": { "xray": { "auto_instrumentation": false, "sdk_version": "0.16.0", "sdk": "opentelemetry for nodejs" } }, "metadata": { "default": { "otel.resource.telemetry.sdk.name": "opentelemetry", "net.transport": "IP.TCP", "http.flavor": "1.1", "http.route": "/woops", "net.host.port": "", "http.status_text": "NOT MODIFIED", "otel.resource.service.name": "test", "otel.resource.telemetry.sdk.language": "nodejs", "net.host.ip": "::ffff:10.2.44.153", "otel.resource.telemetry.sdk.version": "0.16.0" } },

status (and also content) in the http response block are always 0. The metadata block contains http.status_text which is actually shown in xray, but http.status_code is not in there.

awssandra commented 3 years ago

Interesting!

So the X-Ray components only handle the X-Ray trace ID generator and propagation - all data is recorded in the OpenTelemetry format, sent to the Collector, where it's converted in the X-Ray exporter. The code for the exporter is hosted upstream, so it's over in the opentelemetry-collector-contrib repo.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/exporter/awsxrayexporter/translator/http.go#L57

Seems like this should be working, but is not :) for whatever reason. Give us a min to take a look.

In the meantime, can you confirm your dependency versions for the OpenTelemetry and X-Ray components? There shouldn't be anything related, but a mismatch between the Collector and OpenTelemetry core packages in the various stages of the spec implementation might cause issues. So, just want to rule that out.

Thanks! Sandra

niekname commented 3 years ago

These are the versions that I use at the moment:

"@aws/otel-aws-xray-id-generator": "0.13.1",
"@aws/otel-aws-xray-propagator": "0.13.0",
"@opentelemetry/core": "0.16.0",
"@opentelemetry/exporter-collector-grpc": "0.16.0",
"@opentelemetry/metrics": "0.16.0",
"@opentelemetry/node": "0.16.0",
"@opentelemetry/plugin-express": "0.13.1",
"@opentelemetry/plugin-http": "0.16.0",
"@opentelemetry/plugin-https": "0.16.0"
niekname commented 3 years ago

I just updated to opentel 0.18 and now the http status code is visible in xray.

these are the versions I use now:

"@aws/otel-aws-xray-id-generator": "0.13.1",
"@aws/otel-aws-xray-propagator": "0.13.0",
"@opentelemetry/core": "0.18.0",
"@opentelemetry/exporter-collector-grpc": "0.18.0",
"@opentelemetry/instrumentation": "0.18.0",
"@opentelemetry/metrics": "0.18.0",
"@opentelemetry/node": "0.18.0",
"@opentelemetry/plugin-express": "0.14.0",
"@opentelemetry/plugin-http": "0.18.0",
"@opentelemetry/plugin-https": "0.18.0"