Need option to propagate some headers 'as-is' in interceptors/handlers

[eoliphan]

Hi, first great lib, really takes a lot of the pain out of using otel w/ clojure.

Anyway, I'm using Clj-Otel on AWS w/ datomic ions. So http requests are basically:

APIGW (w/ xray) -> AWS Lambda (w/ xray) -> Datomic Cloud JVM (instrumented w/ otel) -> Datomic Ion adapter (puts ring headers on request) -> my handler func w/ pedestal router (clj-otel integrated)

I'm having a problem with trace propagation, as in this config, APIGW is going to stick an XRay X-Amzn-Trace-Id header on the request, that gets carried along to the handler. But traces stop at the lambda, and this appears to be due to the fact that the span doesn't have the X-Amzn-Trace-Id attribute, therefore the Xray propagator isn't property linking stuff up.

The :captured-request-headers option for the (server-span-interceptors) function works great for generically adding additional headers as attributes with that http.request.header prefix. But need to be able to specify similar except that the specified headers are added as attributes 'as-is'. Or perhaps just an option for a generic function that could be used to further tweak the final map of attributes that end up on that :io.opentelemetry/server-request-attrs pedestal context attribute

[steffan-westcott]

What you are describing utilises a fundamental concept called context propagation in OpenTelemetry terms. By default, both OpenTelemetry Java and clj-otel have enabled propagators for W3C Trace Context and W3C baggage header, though other propagators are available. There is an AWS X-Ray Trace Header protocol propagator, but it is not included nor enabled by default.

The method to include and enable the AWS X-Ray Trace Header protocol propagator depends on how your application is run.

• If the application is run with the instrumentation agent, ensure the system property otel.propagators comma delimited list of propagators includes xray
• If the application is run with the autoconfigure SDK extension, set the otel.propagators system property as above and include the Java library dependency io.opentelemetry.contrib/opentelemetry-aws-xray-propagator
• If the application is run with a programmatically configured SDK, include the Clojure dependency com.github.steffan-westcott/clj-otel-contrib-aws-xray-propagator and ensure propagators option to init-otel-sdk! includes (steffan-westcott.clj-otel.propagator.aws-xray/aws-xray-propagator)

The captured-request-headers option is unrelated to your use case. That option controls which HTTP headers are copied as attributes into the server span. It has no bearing on context propagation at all.

[eoliphan]

Ok, understand. So, i'm actually using AWS' implementation of both the agent and the collector. Per their agent docs, it seems like xray should already be in the list. "Out of the box, it propagates traces using all of W3C Trace Context, B3, and X-Amzn-Trace-Id.". But going to double check. If it is enabled, one problem might be that this 'http request' is actually coming in via this datomic/lambda adapter. So wondering if agent doesn't 'know' about the code in this case that needs to be instrumented.

[steffan-westcott]

I have no experience with this myself, but I know that support for OpenTelemetry with AWS Lambda is typically implemented using a Lambda Layer. There is an official AWS ADOT for this : https://aws-otel.github.io/docs/getting-started/lambda