Documentation Request Clarification (For custom tag and global config span tags)

[lorenzoash]

We are reviewing your documentation on adding custom tags and global config span tags in order to surface additional details and while there is some simple code provided here, it isn't entirely clear where certain changes are to be made in the XML to generate these tags.

For some of our HTTP and DB calls we are looking to surface some additional tags. So while the spans are being generated correctly there is just some added context we would like to create.

and then we are also doing a custom flow into an IBM mainframe and for this we are trying to get the appropriate basic tagging to be provided for this so it will treat it as if it is a DB call. Namely to surface basic things like "span.kind" = client, and "db.name" = IBM so it will allow Splunk APM to create inferred services for these as a DB call.

for the custom tags, it seems like if you add this to the general OTEL config xml it will just put this tag on all flows/spans which isn't helpful when we just want one single tag added for a specific flow. And for the global config span tags, it looks like this would be more in line with getting a tag added to just one specific flow, however it's not clear where you set the "Remote_Request_configuration.otel.peer.service=my_remote_api" in your example.

[jefair2]

Hello, just to add, it looks like one piece we were missing and not quite understanding was that with a multi-XML project you need to add at the top of each XML file the Namespace references and Schema links. Otherwise it will throw an error that

The prefix "opentelemetry" for element "opentelemetry:add-custom-tags" is not bound.

Once we resolved that issue by adding the schema information at the top, we injected the "add-custom-tags" into the code, and it threw an error on compiling:

[Application] There was '1' error while parsing the given file 'flows/get-customer-details.xml'. - Exit code: 1, Restart count: 2

So it didn't like where we added the XML into the code for some reason. We are trying to troubleshoot that further.

We are still unclear where to try and add the Global tags.

[manikmagar]

Thank you @lorenzoash @jefair2 for reaching out here. Let me see if I can clarify it here -

As in the Custom Tags documentation,

In addition to all the trace attributes captured by the module, it is possible to add custom tags to the current trace using an operation opentelemetry:add-custom-tags.

opentelemetry:add-custom-tags is a new operation on the connector. You would use that as any other operation in mule within flow/sub-flow.

it seems like if you add this to the general OTEL config xml it will just put this tag on all flows/spans

Any tags added using this operation will get added to the current trace's parent span representing that request processing, and NOT on any other flow spans or request spans. If that flow is invoked 4 times with 4 different orderId, the 4 traces will have their own orderNumber tag with a value from attributes. So, custom tags are added at the request level.

NOTE: With this, you cannot add a custom tag using this operation to a specific connector operation or processor.

Global Config Span tags

... is a way to add additional tags to the spans of components that use global configuration elements.

For how to do it part -

To add a tag peer.service=my_remote_api to http:request 's span, you can set following system property on mule runtime - Remote_Request_configuration.otel.peer.service=my_remote_api

There are multiple ways to set system properties for mule runtime like adding in mule wrapper conf, setting on runtime manager etc. See Mule Docs.

@jefair2

it looks like one piece we were missing and not quite understanding was that with a multi-XML project you need to add at the top of each XML file the Namespace references and Schema links.

I don't know how your mule code is structured. In general, for any mule XML to use any operation or source or component, corresponding XML namespace must be declared at the top. It is a normal Mule behavior. Do you mean you had to add it to mule XML configurations that do not have any usage of opentelemetry:add-custom-tags operation?

So it didn't like where we added the XML into the code for some reason.

Could you add full error details here?

[jefair2]

We are still trying to work through getting the tags to work, but the issue I was commenting on appears to be associated with not running the latest version of 1.2, as we didn't realize this feature was added in that version. Once we cleared through that it seemed to resolve the issues we were facing.

We focused in on the global config span tags, since that seems more conducive to being able to set the values directly based on the description.

So we have this HTTP Request configuration we were going to use just to test this. You can see if has a "name" defined

So we set the Global property using the same formatting suggested in the docs as follows:

And then redeployed and ran some tests:

On this you can see the "inferred span" is still going with the default name of the HTTP URL instead of the override which should take preference for "peer.service" if this was getting applied.

And you can see there is no "peer.service" tag showing up in the results. Note that you can tell this is for sure the same config stanza since the tag that's automatically put in for "mule.app.processor.confRef" points to the same "TSOCreds_HTTP_Request_configuration" we were setting.

So what are we missing here on trying to get this configured?

[manikmagar]

@jefair2 your HTTP Config seems correct. However, I think there may be some confusion with what peer.service can do on Splunk side.

On this you can see the "inferred span" is still going with the default name of the HTTP URL instead of the override which should take preference for "peer.service" if this was getting applied.

As per Splunk documentation peer.service is used to identify name of the inferred service and not the name of the span. HTTP URL is name of the span and thus is independent of the peer.service.

So, by setting the peer.service, you will get a change in the value you abstracted in your screenshot.

Global Config spans are supported via System Properties. Setting global-property in mule configuration isn't getting into system properties and hence the attribute is not set. You may want to check System Properties for On-Prem or Runtime Manager for setting attribute values.

Also, name of the span is different than span attributes. Name is immutable and cannot be changed once a span is created. However attributes (e.g http.host, http.method etc) is what can possibly be changed using global config span.

May I ask what are you intending to achieve with those attributes?

[jefair2]

Sure, so ultimately the IBM connection we are trying to effectively build that to represent as an RPC or HTTP type connection. Since this is not one of your defined auto-instrumented flows (e.g. HTTP Requests), it is just showing up as a span, but without any of the relevant values captured. So we are ultimately trying to add and/or override certain values captured in that flow.

To test getting it to work at all, we were simply trying to manipulate a different and working HTTP Request as per the example just to try and figure out how to get the value to override before trying to apply that functionality outside of your documented route and see about manipulating the IBM requests.

As for the purpose of peer.service I am aware of what it is intended for, If it was working correctly on the example "retriveTSOCredentials" call it would override the inferred service from the HTTP URL to the value we tried to specify (In this case "test").

Ultimately we need a way to set properties for any flow according to the OpenTelemetry Tracing Specifications: https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions

As these have very specific meanings, the properties need to be put into the spans as they are listed in the specification, as that is what any destination product (in this case Splunk APM, but I would imagine the other products expect similar behavior) is expecting to see. Hopefully this helps?

[manikmagar]

the IBM connection we are trying to effectively build that to represent as an RPC or HTTP type connection

"IBM connection" -> Is this a different mule connector? OR just over HTTP APIs?

Since this is not one of your defined auto-instrumented flows

If it is a connector, then maybe supporting it with its own processor is the correct way of collecting attribute data for it.

Here are screenshots of our demo application using global config spans for Splunk -

Properties configuration when running from Anypoint Studio -

Global Config spans are supported via System Properties. Setting global-property in mule configuration isn't getting into system properties and hence the attribute is not set (see #85 ). You may want to check System Properties for On-Prem or Runtime Manager for setting attribute values.

Trace with inferred spans:

NOTE: Even though peer.service was sent and used, Splunk is still Service maps make it more clear for when peer.service is used. For example, the representation of map for above trace -

Regarding semantic conventions, all the span data (name, kind, and attributes/tags) collected by this module are as per the specifications in those conventions and are put into the span representing that component. Any OpenTelemetry-compliant APM backend should be able to accept and process those.

For adding/modifying module collected data, only attributes are possible to be overridden by users using global config (starting v1.2.0) but they are limited to be at a particular configuration level. If two operations are sharing the same global config then both spans will get those attributes. So, it is best to be used it that way.

If you haven't tried setting global config tags with System properties, please try it that way and you should see the service name changing on the Splunk side.

[jefair2]

We managed to finally get it to work correctly by setting it as a VM argument. This is definitely clunky but it's better than nothing for when you need to build a custom connector that isn't something you support yet.

That said, yes the IBM Connector is a mulesoft connector that can be added. It isn't super commonly used by everyone, but is something we are using here. It's for connections into IBM Mainframe.

We also have some other things that aren't quite working as intended, but we are going to do some additional digging and try and circle back in a few days (likely in a different issue as it will be unrelated to this), in the meantime, what I would suggest is to expand your documentation on the Global tagging to include some of the details provided here in this thread as it made the difference in being able to get it to work correctly.