Closed vaibhhavv closed 1 month ago
Pinging code owners:
processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
See Adding Labels via Comments if you do not have permissions to add labels yourself.
From what I can tell the keep_keys
editor is doing a strict match check, not accounting for regex pattern matching. keep_matching_keys
may work, but it only takes a single regex to match. I'll defer to code owners to know if there's an option I'm missing here, or if this should be added.
From what I can tell the
keep_keys
editor is doing a strict match check, not accounting for regex pattern matching.keep_matching_keys
may work, but it only takes a single regex to match. I'll defer to code owners to know if there's an option I'm missing here, or if this should be added.
Hi! When using the keep_matching_keys
function, you could try out the following expression to achieve the same effect:
^(aaa|bbb|ccc).*
This way you do not have to use an array of expressions, which is not supported by the keep_matching_keys
function
I did some further checking, and recognized that when using resource
as a context
none of the attributes were dropped. However, when providing span
as the context, i was able to delete all unmatched attributes in my spans and the resource using the following config:
trace_statements:
- context: span
statements:
- keep_matching_keys(attributes, "^(aaa|bbb|ccc).*")
- keep_matching_keys(resource.attributes, "^(aaa|bbb|ccc).*")
Regarding resource
seemingly not being a valid value for the context
I'll also have to defer to the code owners if this is intended or not, but I hope the suggestion above is of some help to you @vaibhhavv .
One more note: I just checked, and the keep_matching_keys
function is available since v0.103.0, so to use that you would have to upgrade from the version you are currently using
Many thanks @crobert-1 , @bacherfl for your inputs.
We made it! And yes, the above config with context: span
is perfect for the use case. ❤️
I have a few following questions:
Now, Let's say we want to keep another attribute xyz.pqr
with the above ones and delete all others.
What is the best way you suggest to do that?
a) Add another keep_matching_keys
statement below the config
b) Add the same statements
Note: here is xyz.pqr
is a standalone attribute we want to mention in the keep_matching_keys
and others we configured as regex.
As you mentioned @bacherfl , context: resource
is not dropping attributes, same for us and we use context: span
.
But we observe, In spans we are receiving Span Attributes, Resource Attributes, and Events as well.
Currently, we are dropping data from the first two but as you can see in the screenshot the attribute message
is not getting dropped from events.
We also used context: spanevent
but after that, no attributes were dropping from any of them. Do we have some other way to handle that?
Hi @vaibhhavv, I can also answer a few of your questions. First, note that you can use multiple different contexts within the same transform processor configuration, you're not limited to only a single context.
For your specific questions:
keep_matching_keys
will remove any keys that aren't in the regex, so you will need to add it to the regular expression you use.Here's a sample config to illustrate the above points:
trace_statements:
- context: resource
statements:
- keep_matching_keys(attributes, "^(aaa|bbb|ccc).*")
- context: span
statements:
- keep_matching_keys(attributes, "(^xyz.pqr$)|(^(aaa|bbb|ccc).*)")
- context: spanevent
statements:
- delete_key(attributes, "message")
Thanks @evan-bradley for your input here. I got what I was looking for. ❤️
Component(s)
processor/transform
What happened?
Description
I am utilising transform processor and receiving traces on my otel collector. Use case is I want to drop all other attributes (both span & resource attributes) coming in that traces except few whose initials start with 'aaa.' , 'bbb.', 'ccc.'. I configured the transform processor but it is not behaving according to the expectation. It is not dropping any attribute.
Steps to Reproduce
Expected Result
All attributes should get dropped except the ones which starts with 'aaa.' , 'bbb.' , 'ccc.'
Actual Result
No attributes are getting dropped
Collector version
0.102.1
Environment information
Environment
OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
Log output
No response
Additional context
No response