OTEL plugin sends incoming W3C tracestate and Instana tracing HTTP headers out as received

[sriemer]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

Kong 3.5.0

Current Behavior

Receiving Instana + W3C tracing headers like

X-INSTANA-L: 1
X-INSTANA-T: 2696d845a4b364f5
X-INSTANA-S: ecbea6329e46a665
traceparent: 00-00000000000000002696d845a4b364f5-ecbea6329e46a665-01
tracestate: in=2696d845a4b364f5;ecbea6329e46a665

Sending out tracing headers like

X-INSTANA-L: 1
X-INSTANA-T: 2696d845a4b364f5
X-INSTANA-S: ecbea6329e46a665
traceparent: 00-00000000000000002696d845a4b364f5-d666cccb26e1ba7e-01
tracestate: in=2696d845a4b364f5;ecbea6329e46a665

Next Instana tracer in the chain picks up span ID from X-INSTANA-S, fallback: tracestate in member, only then traceparent. So on the Instana UI it appears as if spans are directly sent from client app to server app without Kong.

Expected Behavior

Would be nice to be able to add support for Instana headers or at least filtering out Instana headers and the tracestate header.

Is that possible with custom LUA code?

Does "customize OTel spans as a developer" (https://docs.konghq.com/hub/kong-inc/opentelemetry/#customize-opentelemetry-spans-as-a-developer) work for this?

So expecting to see just header traceparent here with the new span ID:

traceparent: 00-00000000000000002696d845a4b364f5-d666cccb26e1ba7e-01

Steps To Reproduce

1. clone our demo repo kong-tracing -> it is private for now
2. get access to Instana UI with an own agent key and configure file .env
3. run docker-compose build && docker-compose up
4. get the gateway network interface used by docker-compose via ip -s a | grep $(docker inspect $(docker ps | grep kong-tracing_kong | cut -d' ' -f1) | grep "Gateway.*[0-9\.]\+" | grep -o "[0-9\.]\+")
5. take a tcpdump via tcpdump -s 0 -i $BRIDGE_INTERFACE -w kong-tracing-demo.pcap
6. analyze the headers with wireshark

OTel plugin config:

plugins:
- name: opentelemetry
  config:
    endpoint: "http://instana-agent:4318/v1/traces"
    resource_attributes:
      service.name: "kong-otel-demo"

Anything else?

OS: Linux x86_64

[sriemer]

Okay, found a workaround for this issue. :tada: :tada: :tada: :smiley:

First working custom LUA code:

-- Modify the root span
local root_span = kong.tracing.active_span()
-- Use NGINX request API directly
h = ngx.req.get_headers()
local span_id_in = h["x-instana-s"]
local tracestate_in = h["tracestate"]
if span_id_in then
  root_span:set_attribute("headers.x-instana-s", span_id_in)
  root_span:set_attribute("headers.dropping-x-instana-s", "true")
  ngx.req.set_header("x-instana-s", "")
else
  root_span:set_attribute("headers.no-x-instana-s", "true")
end
if tracestate_in then
  root_span:set_attribute("headers.tracestate", tracestate_in)
  root_span:set_attribute("headers.dropping-tracestate", "true")
  ngx.req.set_header("tracestate", "")
else
  root_span:set_attribute("headers.no-tracestate", "true")
end

My demo uses the declarative YAML config and Kong without database. So I have to put the code into kong.yml. Unfortunately, the Kong post-function method access does not allow reading a LUA file in there, expects an equals sign '=' and an array in a single line. So I have to inject that code into a single line by:

• dropping comment lines
• escaping double-quotes "
• replacing newlines with ;
• injecting all this via envsubst and an ${OTEL_LUA_CODE} variable

kong.yml.in part:

plugins:
- name: opentelemetry
  config:
    endpoint: "http://instana-agent:4318/v1/traces"
    resource_attributes:
      service.name: "kong-otel-demo"
- name: post-function
  config:
    access: ["${OTEL_LUA_CODE}"]

[sriemer]

So the last remaining things that I'm requesting are:

1. that it will be documented that incoming trace headers are forwarded as is (except traceparent) and
2. that it is put into the LUA code example how to get, set, and drop those trace headers

Thanks in advance.

[samugi]

Hi @sriemer thank you for opening this issue.

X-INSTANA-* headers are currently not supported by Kong; that's why they are ignored and their content remains unchanged in the request. This is true for any header format that is not yet supported.

The good news is that the tracing headers propagation module has just been reworked to allow more configurability of tracing headers propagation and also to make it easier to add support for new headers. The feature has not been released yet and as of today it is only available in development branches/images. Documentation updates are still WIP but will include examples of tracing headers configurations, we will make sure to cover the points you highlighted.

If you'd like to check out what's changed you can have a look in the master branch:

• the propagation schema options will be available for configuration in tracing plugins, like OpenTelemetry
• Extractors and injectors for all supported header types now exist in dedicated files, see for example: w3c's extractor and injector which define how each header format must be handled

In order to add support for a new header type so that it can be configured to be extracted, injected or cleared with no need for additional custom Lua code it is enough to add dedicated extractor and injector modules. If you would like to open a Pull Request to add Instana headers support I will be happy to review it. Otherwise, I will take note of this requirement for potential consideration in the future.

Thank you

[github-actions[bot]]

This issue is marked as stale because it has been open for 14 days with no activity.

[github-actions[bot]]

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team

[sriemer]

@samugi Thanks so much for that great work. :pray: That will help us a lot. I'll discuss this with our product management and check when we can do the Instana support for this.