[Bug]: DatadogTraceHandler Attempt to Call Extractor Failed

[mfischler]

Kong Version

3.6.1

Plugin Version

0.2.0-1

On which environment your Kong instance is running?

Kubernetes

Plugin Configuration

- name: ddtrace
    config:
      service_name: SERVICE_NAME
      injection_propagation_styles:
        - datadog
      extraction_propagation_styles:
        - datadog
      header_tags:
        - header: X-Gateway
          tag: gateway
      resource_name_rule:
        - match: /route/\w*/info
          replacement: /route/?/info

DD_ENV:
  valueFrom:
    fieldRef:
      fieldPath: metadata.labels['tags.datadoghq.com/env']
DD_SERVICE:
  valueFrom:
    fieldRef:
      fieldPath: metadata.labels['tags.datadoghq.com/service']
DD_VERSION:
  valueFrom:
    fieldRef:
      fieldPath: metadata.labels['tags.datadoghq.com/version']
DD_AGENT_HOST:
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
DD_TRACE_AGENT_PORT: 8126
DD_TRACE_AGENT_URL: http://$(DATADOG_AGENT_HOST):$(DATADOG_TRACE_AGENT_PORT)
DD_TRACE_PROPAGATION_STYLE_INJECT: '["datadog"]'
DD_TRACE_PROPAGATION_STYLE_EXTRACT: '["datadog"]'

What happened?

[error] 1322#0: *2498 [kong] handler.lua:360 [ddtrace] tracing error in DatadogTraceHandler:header_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while reading response header from upstream, client: 192.168.x.x

[error] 1322#0: *2498 [kong] handler.lua:379 [ddtrace] tracing error in DatadogTraceHandler:body_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while sending to client, client: 192.168.x.x

[error] 1322#0: *2498 [kong] handler.lua:379 [ddtrace] tracing error in DatadogTraceHandler:body_filter: ...local/share/lua/5.1/kong/plugins/ddtrace/propagation.lua:41: attempt to call local 'extractor' (a nil value) while sending to client, client: 192.168.x.x

Type: Kong DB-Less Kubernetes: 1.28

I suspect that DDTrace Plugin does not have access due to permissions. Would 'kong.ctx.shared' work instead?

[mfischler]

56

I believe the error is from a misconfiguration with the Extraction/Injection Propagation Styles.

Sample correct configuration: [kong] handler.lua:203 [ddtrace] DATADOG TRACER CONFIGURATION - { ["environment"] = development,["service"] = SERVICE-NAME,["version"] = x.x.x,["extraction_propagation_styles"] = { [1] = datadog,[2] = tracecontext,} ,["injection_propagation_styles"] = { [1] = datadog,[2] = tracecontext,} ,["agent_url"] = http://hostIP:8126,["__id__"] = 28,} , context: init_worker_by_lua*

Is there a correct format to pass these values as an environment? If you do not specify this configuration, the default will be used correctly but will result in excess log messages looking for Vault references for the propagation styles.

[mfischler]

I am not sure if you can set a lua table through an environment variable. Can I recommend eliminating the ability to use an environment variable for configuring propagation systems and require using the configuration file.

ddtrace_conf = {
    __id__ = conf["__seq__"],
    service = get_env("DD_SERVICE") or conf.service_name or "kong",
    environment = get_env("DD_ENV") or conf.environment,
    version = get_env("DD_VERSION") or conf.version,
    agent_url = get_env("DD_TRACE_AGENT_URL") or conf.trace_agent_url or agent_url,
    injection_propagation_styles = conf.injection_propagation_styles,
    extraction_propagation_styles = conf.extraction_propagation_styles,
}

[dmehala]

Hi,

It's a duplicate of #56. Consider DD_TRACE_PROPAGATION_STYLE_INJECT and DD_TRACE_PROPAGATION_STYLE_EXTRACT as not production ready. Instead use the corresponding config field, respectively, injection_propagation_styles and extraction_propagation_styles (which you already did in the config, so you don't need the env variables).

Thanks.