[Bug]: DD_TRACE_PROPAGATION_STYLE_INJECT, DD_TRACE_PROPAGATION_STYLE_EXTRACT environments are not work

[jjuakim]

Kong Version

3.6.1

Plugin Version

0.2.0-1

On which environment your Kong instance is running?

Kubernetes

Plugin Configuration

"config": {
  "agent_host": "${DATADOG_AGENT_HOST},
  "agent_endpoint": null,
  "trace_agent_port": 8126,
  "trace_agent_url": null,
  "injection_propagation_styles": [
    "datadog",
    "tracecontext"
  ],
  "extraction_propagation_styles": [
    "datadog",
    "tracecontext"
  ],
  "initial_samples_per_second": 100,
  "initial_sample_rate": null,
  "max_header_size": 512,
  "static_tags": null,
  "resource_name_rule": null,
  "generate_128bit_trace_ids": true,
  "service_name": "my-kong",
  "header_tags": null,
  "environment": "develop"
}

What happened?

I updated to the latest version, but this warning messages appeared.

2024/04/17 08:33:19 [warn] 1329#0: *51712 [kong] vault.lua:1255 could not retrieve value for reference {vault://env/DD_TRACE_PROPAGATION_STYLE_INJECT} (could not get value from external vault (no value found)), context: ngx.timer
2024/04/17 08:33:19 [warn] 1329#0: *51712 [kong] vault.lua:1255 could not retrieve value for reference {vault://env/DD_TRACE_PROPAGATION_STYLE_EXTRACT} (could not get value from external vault (no value found)), context: ngx.timer 

so I added DD_TRACE_PROPAGATION_STYLE_INJECT and DD_TRACE_PROPAGATION_STYLE_EXTRACT environments to k8s like that:

spec.template.containers[].env ...

template:
  spec:
    containers:
       - name : my-kong
          env:
          - name: DD_TRACE_PROPAGATION_STYLE_INJECT
             value: "datadog,tracecontext"
           - name: DD_TRACE_PROPAGATION_STYLE_EXTRACT
             value: "datadog,tracecontext"

so I expected the problem to be resolved, but I still received the following error message:

2024/04/19 06:03:17 [error] 1327#0: *61661 [kong] handler.lua:333 [ddtrace] tracing error in DatadogTraceHandler:access: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value), client: 10.117.98.94, server: kong, request: "POST /api/foo/bar HTTP/1.1", host: "some-host:8080", referrer: "referrer-url", request_id: "2f00736f775fce594504073fa8a8de4b"
2024/04/19 06:03:17 [error] 1327#0: *61661 [kong] handler.lua:360 [ddtrace] tracing error in DatadogTraceHandler:header_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while reading response header from upstream, client: 10.117.98.94, server: kong, request: "POST /api/foo/bar HTTP/1.1", upstream: "https://upstream-host:443/api/foo/bar", host: "some-host:8080", referrer: "referrer-url", request_id: "2f00736f775fce594504073fa8a8de4b"
2024/04/19 06:03:17 [error] 1327#0: *61661 [kong] handler.lua:379 [ddtrace] tracing error in DatadogTraceHandler:body_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while sending to client, client: 10.117.98.94, server: kong, request: "POST /api/foo/bar HTTP/1.1", upstream: "https://upstream-host:443/api/foo/bar", host: "some-host:8080", referrer: "referrer-url", request_id: "2f00736f775fce594504073fa8a8de4b"
2024/04/19 06:03:17 [error] 1327#0: *61661 [kong] handler.lua:379 [ddtrace] tracing error in DatadogTraceHandler:body_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while sending to client, client: 10.117.98.94, server: kong, request: "POST /api/foo/bar HTTP/1.1", upstream: "https://upstream-host:443/api/wrong-deposit/v1/wrong-addresses/statistics", host: "some-host:8080", referrer: "referrer-url", request_id: "2f00736f775fce594504073fa8a8de4b"

What did I miss or did I configure it incorrectly?

Is there a way to check if the correct values are in the vault? I want to solve this problem.

[dmehala]

Hi @jjuakim

Thank you for reporting this bug. It's a mistake on my end, DD_TRACE_PROPAGATION_STYLE_* are not parsed, resulting in this error.

Since datadog,tracecontext is the default value for extraction/injection propagation styles, there's no necessity to explicitly specify them. However, if you still need to configure extract/injection propagation styles, I recommend using the corresponding plugin configuration fields injection_propagation_styles and extraction_propagation_styles.

In case you need assistance to configure a plugin in a k8s environment, here is an example from Kong's documentation.

[jjuakim]

@dmehala I hope it will be fixed soon 😄 I'll use configuration filed instead of using environments variables.
thank you for answer!

[mfischler]

Having the same issue. I am not sure if you can pass a lua table object using an environment variable. I modified source and rebuilt. See #57