Lambda is implementing PassThrough mode which allows customers to send traces through instrumented lambda functions without those functions being traced. The way this is done is by changing the trace header structure from Root=...;Parent=...;Sampled=1 in the active tracing case to Root=... in the PassThrough case. As such, our SDK needs to be updated to handle this.
Changes
Update logic to use no-op segment if parent and sampled are missing (ie. PassThrough mode)
Had to use SuppressWarnings("nullness") because TraceHeader.getRootTraceId() returns an @Nullable TraceId but Segment.noOp() expects a @NonNull TraceId
Update all trace header propagations to only propagate Root if we're not actively sampling
Not propagating Parent and Sampled is fine because Parent will be re-generated in the next traced context and missing Sampled will be treated as not actively sampling anyways
We can't propagate only one of these, it needs to be either both or neither
Updated unit tests
Testing
Ran many extensive tests using v2.16.0, v2.17.0, and these changes to compare the outcomes. In these tests I configured 3 lambda functions to interact with each other ALL instrumented with the new changes:
Lambda A: Calls Lambda B
Lambda B: Calls Lambda C
Lambda C: Calls GetAccountSettings
The trace headers in the below cases are from logging System.getenv("_X_AMZN_TRACE_ID") in the lambda functions themselves
The following are some important cases that prove the functionality of the code change
Case 1.a: Lambda A Active → Lambda B PassThrough → Lambda C Active
Lineage is NOT propagated or received by any of these functions
Lambda B has an inferred node due to the invocation in Lambda A which is actively traced, this is expected behaviour and not controlled by the SDK
Lambda B function is being appropriately suppressed aside from the previous note
Lambda C invocation is appropriately connected to the Lambda A function
Case 2: Lambda A PassThrough → Lambda B Active → Lambda C PassThrough
Lineage is NOT propagated or received by any of these functions
Lambda B is the start of the trace as expected
Lambda C invocation is still attached to Lambda B as an inferred node (similar to case 1)
Lambda C's GetAccountSettings call IS TRACED; this is expected behaviour because we entered Active tracing mode in Lambda B and this is meant to propagate to downstream services. If the downstream service has its own version of passive tracing/X-Ray integration, it should be configured as such, but PassThrough functionality should only apply to Lambda functions and not the downstream calls they trigger unless the entire pathway has been in PassThrough
Case 3.a: Calling Lambda B PassThrough → Lambda C PassThrough
Lineage not propagated
No trace, proves statement above that a complete trace entirely in PassThrough mode will not trace the final GetAccountSettings call since we never entered Active tracing mode.
Issue
Lambda is implementing PassThrough mode which allows customers to send traces through instrumented lambda functions without those functions being traced. The way this is done is by changing the trace header structure from
Root=...;Parent=...;Sampled=1
in the active tracing case toRoot=...
in the PassThrough case. As such, our SDK needs to be updated to handle this.Changes
SuppressWarnings("nullness")
becauseTraceHeader.getRootTraceId()
returns an@Nullable TraceId
butSegment.noOp()
expects a@NonNull TraceId
Root
if we're not actively samplingTesting
Ran many extensive tests using v2.16.0, v2.17.0, and these changes to compare the outcomes. In these tests I configured 3 lambda functions to interact with each other ALL instrumented with the new changes:
GetAccountSettings
The trace headers in the below cases are from logging
System.getenv("_X_AMZN_TRACE_ID")
in the lambda functions themselvesThe following are some important cases that prove the functionality of the code change
Case 1.a: Lambda A Active → Lambda B PassThrough → Lambda C Active
Case 1.b: Same thing but Lambda A Active sets
Sampled=0
Case 2: Lambda A PassThrough → Lambda B Active → Lambda C PassThrough
GetAccountSettings
call IS TRACED; this is expected behaviour because we entered Active tracing mode in Lambda B and this is meant to propagate to downstream services. If the downstream service has its own version of passive tracing/X-Ray integration, it should be configured as such, but PassThrough functionality should only apply to Lambda functions and not the downstream calls they trigger unless the entire pathway has been in PassThroughCase 3.a: Calling Lambda B PassThrough → Lambda C PassThrough
GetAccountSettings
call since we never entered Active tracing mode.Case 3.b: Calling Lambda B Active → Lambda C Active
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.