OTEL plugin: No way to set variable values to resource attributes

[sriemer]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

Kong 3.5.0

Current Behavior

All configured OTel resource attributes are taken from config as plain strings. I do not see any possibility to set a variable value such as the current host name for example as resource attributes.

For span to OTel infrastructure linking on the Instana UI I need special OTel resource attributes such as host.name, process.pid, and container.id.

But no type of variable is resolved.

Expected Behavior

If I put

      resource_attributes:
        service.name: "kong-otel-demo"
        host.name: $host

into the OTel plugin config, then I would expect this to be resolved as the same value as from span attribute http.host. But it sends out "$host" instead. In my demo repo, the hostname of my Kong container would be "kong-gateway".

Steps To Reproduce

1. clone our demo repo kong-tracing -> it is private for now
2. get access to Instana UI with an own agent key and configure file .env
3. try to configure Kong via kong.yml for resource attributes with variable values
4. run ./run_kong_demo.sh # based on docker-compose
5. check Instana agent trace logs for the span JSON data that gets sent out

Anything else?

Replacing the value of resource attributes with custom LUA code would be nice as well. I imagine something like:

local root_span = kong.tracing.active_span()
root_span:set_resource_attribute("host.name", ngx.var.host)

But I get an error "attempt to call method 'set_resource_attribute' (a nil value)".

Or just putting attributes where they belong to would be nice as well:

local root_span = kong.tracing.active_span()
root_span:set_attribute("resource.host.name", ngx.var.host)

Right now this adds a span attribute "resouce.host.name" instead of resolving this to a resource attribute "tags":{"http.host":"kong-gateway" ... ,"resource":{"host.name":"kong-gateway",...}

[sriemer]

@samugi Are you maybe available to look into this? TIA

[samugi]

@sriemer thank you for opening this issue.

The behavior you describe is expected: there is currently no support for parsing OpenTelemetry's resource_attributes as lua code. I've noted this and we may consider it based on interest. However, if you decide to submit a PR, I'll happily review it to expedite this!

Regarding your additional comments in the "Anything else?" section, the tracing PDK currently only allows setting span attributes, resource attributes are not part of a span's structure, see for example: https://github.com/open-telemetry/opentelemetry-proto/blob/main/examples/trace.json , so it would be required adding new support for this kind of customization in the tracing PDK.

[sriemer]

Thanks for your reply and your offer. I might come back to it. But for now I have issues to get the container ID due to sandboxing. So I'm looking for solutions in Instana agent code right now.

[chronolaw]

@sriemer, do you think that this issue is solved? Could you close it?

[garciasdos]

This is a common issue. I need support modifying OpenTelemetry's resource attributes too. For example, the New Relic OTEL agent asks for more information to be added on OpenTelemetry spans (https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/view-your-data/opentelemetry-transactions-page/)

[StarlightIbuki]

@Water-Melon I remember that you've proposed a feature of variables in configuration. Do you have any idea?

[sriemer]

What I did on our end is to go for the resource. span attribute prefix and to move those to resource attributes in the OpenTelemetryMapper of our Instana agent. I detect Kong from the scope name which is kong-internal in this case. Of cause this is only a workaround. I'm also able to find the Kong pid in the root pid namespace. But then finding out if that is the correct Kong with the correct service.instance.id is tricky. Would be really better if the Kong OTel plugin would read from /proc/$pid/mountinfo and send the current container ID within spans.

[SuzyWangIBMer]

Is this feature request still on going? Any plan to support it in the near future? @samugi @Water-Melon

[StarlightIbuki]

I think there are 2 features requested in this ticket: 1 for nested attributes and 1 for dynamic sources of custom attributes. At least the latter should be simple: we can make resource_attributes's value referenceable. Any idea? @samugi

[samugi]

@StarlightIbuki it would require supporting templates in order to render the resource attribute values as lua code, which is something we support in other plugins' fields, but is slightly more complex than just making the field "referenceable". At the moment this is not being worked on, but some tracing modules are being reworked as part of different features. We might be able to squeeze this in too, I will follow up with an update if/when that happens.