HTTP Baggage is not propagated to the context for `flask` applications

[StasEvseev]

Summary of problem

OT baggage is not propagated from HTTP Headers to the context for flask application.

I have a setup with two services, both flask application, upstream and downstream ones. I am running services wrapped in ddtrace and I enable DD_TRACE_PROPAGATION_HTTP_BAGGAGE_ENABLED. When I put baggage in upstream service to the root context using:

from ddtrace import tracer

context = tracer.current_trace_context()
context._set_baggage_item("cluster", "1")

And then I call downstream, I see that HTTP Headers are passed correctly, but context is not enriched with baggage in the context.

context = tracer.current_trace_context()
print(context) # context doesn't have baggage

context = HTTPPropagator.extract(request.headers)
print(context) # context has baggage

After debugging I figured that, everything is there in ddtrace to handle context propagation, it is just mismatch of the headers, expected/actual what is coming to the handler.

So, the whole flow looks like this.

Code which handling context propagation:

def activate_distributed_headers(tracer, int_config=None, request_headers=None, override=None):
    # type: (Tracer, Optional[IntegrationConfig], Optional[Dict[str, str]], Optional[bool]) -> None
    """
    Helper for activating a distributed trace headers' context if enabled in integration config.
    int_config will be used to check if distributed trace headers context will be activated, but
    override will override whatever value is set in int_config if passed any value other than None.
    """
    if override is False:
        return None

    if override or (int_config and distributed_tracing_enabled(int_config)):
        context = HTTPPropagator.extract(request_headers)

request_headers here are (truncated):

{'HTTP_OT_BAGGAGE_CLUSTER': '1', 'HTTP_X_DATADOG_TRACE_ID': '13270000568603949658', 'HTTP_X_DATADOG_PARENT_ID': '4169471492044558439', 'HTTP_X_DATADOG_SAMPLING_PRIORITY': '1', 'HTTP_X_DATADOG_TAGS': '_dd.p.dm=-0,_dd.p.tid=669ed55a00000000', 'HTTP_TRACEPARENT': '00-669ed55a00000000b8288335acdeba5a-39dcf03b306f1067-01', 'HTTP_TRACESTATE': 'dd=p:39dcf03b306f1067;s:1;t.dm:-0;t.tid:669ed55a00000000'}

Then inside of the HTTPPropagator in the method extract headers gets normalized. What I have is normalised_headers look like (truncated):

{ 'http_ot_baggage_cluster': '1', 'http_x_datadog_trace_id': '13270000568603949658', 'http_x_datadog_parent_id': '4169471492044558439', 'http_x_datadog_sampling_priority': '1', 'http_x_datadog_tags': '_dd.p.dm=-0,_dd.p.tid=669ed55a00000000', 'http_traceparent': '00-669ed55a00000000b8288335acdeba5a-39dcf03b306f1067-01', 'http_tracestate': 'dd=p:39dcf03b306f1067;s:1;t.dm:-0;t.tid:669ed55a00000000'}

so it has a ot-baggage header as http_ot_baggage_cluster.

And what context injection expect is ot-baggage-. See variable _HTTP_BAGGAGE_PREFIX = "ot-baggage-"

See code:

def _attach_baggage_to_context(headers: Dict[str, str], context: Context):
    if context is not None:
        for key, value in headers.items():
            if key[: len(_HTTP_BAGGAGE_PREFIX)] == _HTTP_BAGGAGE_PREFIX:
                context._set_baggage_item(key[len(_HTTP_BAGGAGE_PREFIX) :], value)

So, we would never detect HTTP Baggage with such headers, therefore context is not propagated with baggage.

Which version of dd-trace-py are you using?

ddtrace==2.9.2

Which version of pip are you using?

pip 24.0

Which libraries and their versions are you using?

`pip freeze`

How can we reproduce your problem?

Having two flask application, one is calling another and adding baggage to the context, and on the second one, we check if context has baggage in it.

What is the result that you get?

No baggage on the downstream service.

What is the result that you expected?

Having baggage propagated from HTTP headers.

[emmettbutler]

Thank you for pointing this out, @StasEvseev. We'll look into it.

cc @mabdinur

[StasEvseev]

@emmettbutler @mabdinur Any updates on an issue?