Make ALBs first-class processes in OTel traces

[spencerwilson]

Is your feature request related to a problem? Please describe. When adding OpenTelemetry instrumentation to systems involving HTTP servers, I'd like request traces to contain a span for each process or host the request passes through. It's currently hard to obtain spans for time spent in AWS Application Load Balancers.

Throughout this issue, consider the following system:

// "A -> B" means process A sends an HTTP request to process B

client -> ALB1 -> app 1 -> ALB2 -> app 2

As of November 2021, ALBs are unaware of OpenTelemetry. They're compliant with W3C Trace Context spec in that they pass through traceparent and tracestate unmodified, if they receive them.[^1]

[^1]: I'm actually unsure if header pass-through is sufficient to make a process literally "compliant" with the spec; I've admittedly not run a fine-toothed comb over the spec.

Naturally, the support for X-Ray's tracing header, X-Amzn-Trace-Id, is a bit greater: they add or modify this header.

• X-Ray docs dated December 2018 state that the header value just contains a Root field, and also "Load balancers do not send data to X-Ray, and do not appear as a node on your service map".
• ALB docs dated June 2020 additionally mention a Self field, set (or overwritten, if already defined) when the incoming request already contains an X-Amzn-Trace-Id header.

Helpfully, a value for X-Amzn-Trace-Id is also present in ALB access logs (doc)—although it's unclear if the logged value is the value of the header received by the ALB or sent in the request to the upstream target.

Describe the solution you'd like

I'd love it if ALBs could act like other processes in my system (which is trying to follow recommendations for greenfield OTel systems).

First, make ALBs support the following two configurations:

• Pass through traceparent and tracestate unmodified (status quo)
• Mutate incoming traceparent header: Update parent-id

The definition for the latter can be found under "allowed mutations [to traceparent]" defined in the W3C Trace Context spec's section "3.4 Mutating the traceparent Field". Notably, this mutation effectively trusts the incoming trace context, including sampling decision—trust which can create a DoS vulnerability. As a future enhancement, supporting the "Restart trace" mutation defined in the spec might be a more secure alternative to "Update parent-id". I don't have an opinion on how an ALB should set the sampled flag during "Restart trace" mutation.

Second, do one of the following:

• When the new "Update parent-id" ALB configuration is enabled, add two fields to ALB access log entries: 1) the incoming traceparent header, the 2) outgoing traceparent header. This data would enable out-of-band synthesis of span data via log tailing. From there,
  • AWS provides a receiver plugin for my Collector that tails the log, reading span data from it. Such a receiver would need to somehow coordinate with other Collector instances on the tailing of this log, such that 1) at most one instance is tailing it, 2) ensure that assuming some Collector is online the logs are received and trace data submitted to "traces"-type pipelines within 60 s (arbitrary) of their publication to S3. Maybe the first requirement necessitates the user setting up a DynamoDB table or something, akin to the Terraform S3 backend which demands the user bring their own DynamoDB table to enable requisite distributed locking.
  • AWS provides a standalone program that I run which tails the logs, transforms them, and sends them onto another host/process via OTLP/gRPC. For prior art here see https://github.com/honeycombio/honeyaws/issues/130.
• Some AWS service sends span data via OTLP/gRPC to an OpenTelemetry Collector host of my choosing.
  • If this host is on the public Internet—and maybe even if it's not—I'd want a means of securing the gRPC endpoint on my Collector. This is discussed here. According to the registry, the only implementation of ServerAuthenticator currently is oidcauthextension. Adapting the setup described in this blog post to this feature request: I'd register my AWS account (or some resource within it) as an OAuth 2.0 client application in some OIDC-compliant authorization server I administer, and then in its RPCs to my Collector AWS would present ID Tokens issued to it by the authorization server. Upon receiving RPCs from AWS my Collector would then verify the ID Token and then evaluate authorization based whether the ID Token's aud claim contains some value I specified in my Collector config.

Describe alternatives you've considered

• AWS changes nothing, and I try tailing ALB access logs and deriving the necessary info from them as they are today. I'm not 100% sure but I suspect this isn't viable. In particular, although the Self field on X-Amzn-Trace-Id could maybe be treated as a sort of parent-id by the upstream target, Self is not always defined. Additionally, the propagators used in "app 1" and "app 2" might need to be clever about how they extract trace context from inbound HTTP reqs in order for all 4 spans (one per serving process in the system diagram above) to be correctly linked together, trace context-wise.

Additional context It might be interesting to hear the rationale behind AWS X-Ray's decision to not create X-Ray segments for hops through ALBs:

Load balancers do not send data to X-Ray, and do not appear as a node on your service map.

As a service operator I'm keen on knowing as much as a can about the entire request path.

[sethAmazon]

You may want to ask this question in the instrumentation repos. I do not think this is related to the collector @Aneurysm9 @anuraaga

[spencerwilson]

Thanks for the reply, @sethAmazon ! As I mentioned in the "alternatives you've considered" section, I don't see a way that any amount of open source or third party code could make it possible to achieve the goal stated at the top:

I'd like request traces to contain a span for each process or host the request passes through.

I'd be happy to be proven wrong, though! But if that's true then I'm not sure instrumentations are relevant here.

Rather, my understanding is that the ALB product with its current feature set precludes achieving the stated goal. In that sense this issue is primarily a feature request for the ALB product. aws-otel-collector is relevant only secondarily: The UX here could be really delightful if ALB had certain new features && there was a Collector receiver that consumed those features.

Curious to hear your thoughts, and please let me know if there might be better places to raise this.

[willarmiros]

Hi @spencerwilson,

Thank you for this write up, hopefully I can shed a bit more light on the current tracing integrations with ALB and how best to raise this feature request.

It might be interesting to hear the rationale behind AWS X-Ray's decision to not create X-Ray segments for hops through ALBs

At the time of X-Ray's initial integration with ALB to support the X-Amzn-Trace-Id header, the ALB team did not want to emit a segment to the X-Ray backend because ALB is first and foremost an ultra-low-latency load balancer, and they were concerned with the overhead of creating and emitting spans. However I think it still could be feasible as an opt-in feature.

The best way to raise this request to ALB's attention is by submitting a ticket with AWS Support for the ALB service with this feature request. Customer demand is a big driver of new features, and both the enhanced W3C trace header support and active tracing of ALB (e.g. generating span data instead of only mutating the trace header) are both on our radar, so this demand helps us get them prioritized.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[willarmiros]

@alolita can we have stalebot ignore issues like this feature request? It will still be relevant.

[Aneurysm9]

@willarmiros Would https://github.com/aws-observability/aws-otel-collector/pull/1054 and a Future Features milestone work?

[willarmiros]

@Aneurysm9 yep! Thanks

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.

[willarmiros]

@Aneurysm9 can we keep We should keep this open til we've figured out a way to deal with ALB trace headers

[raimi]

I'd like to add the perspective of someone running a system like described above, where "app 1" and "app 2" are under my control, while "ALB1" and "ALB2" are provided as-is by AWS:

// "A -> B" means process A sends an HTTP request to process B
client -> ALB1 -> app 1 -> ALB2 -> app 2

(I have a log- and trace-collecting tooling in place, that reads all ALB access logs and all logs of all services/apps.)

• ALB1 creates a new X-Amzn-Trace-Id with only Root=, since this is what it does - no way to configure that.
• app 1 ignores the header since it is malformed (missing Self and Sampled)
• app 1 creates a new, valid X-Amzn-Trace-Id header
• ALB2 and app 2 pick that up and pass it along

net result: ALB access logs of ALB 1 are not correlated to the rest of the system (which works).

Please: Please do not wait for AWS to change the behaviour of the ALBs; this will not happen. Please: Just accept a minimal X-Amzn-Trace-Id header and let it start a new trace with the given trace-id. Start a new span, whatever. But keep the value so that the rest of us have a chance of finding their logs :)

Thanks!

P.S.: This has nothing to do with Amazon X-Ray directly. It's just a question of using the de-facto standard X-Amzn-Trace-Id.