TLDR: Hi, I'm trying to filter out noisy spans using the filterprocessor + ottl in the collector config. It appears that I'm not able to do that because the collector version used in that Lambda layer is v0.39.0, which was last released 3 years ago. Filterprocessor requires at least 0.66.0, according to these Honeycomb docs. In terms of my ask - it would be fantastic if AWS could release a version of the AWS Lambda Layer that would support filterprocessor.
git clone https://github.com/kmcquade/otel-lambda-failure-example.git
cd otel-failure-example
make deploy
make invoke
# See the README there for what to look for in the failed invocation
# Delete the stack
make delete
Use case
I am trying to filter out noisy spans using the filterprocessor + ottl on the collector side. We enabled Boto instrumentation in OpenTelemetry but our flame graph is getting littered with a lot of requests. 99% of the time, it is too much information, but sometimes it is helpful to include a span about the AWS API calls. For example, if a call to S3 takes over a certain amount of seconds, or if there's downtime for an AWS API that affected a call.
Here's an example of noisy spans that I want to be able to filter out:
I tried modifying my Lambda Otel collector config to use this:
Describe the question
TLDR: Hi, I'm trying to filter out noisy spans using the filterprocessor + ottl in the collector config. It appears that I'm not able to do that because the collector version used in that Lambda layer is v0.39.0, which was last released 3 years ago. Filterprocessor requires at least 0.66.0, according to these Honeycomb docs. In terms of my ask - it would be fantastic if AWS could release a version of the AWS Lambda Layer that would support filterprocessor.
I created a demo to make it as easy to reproduce the issue: https://github.com/kmcquade/otel-lambda-failure-example. I hope this helps. Just clone it and run these commands:
Use case
I am trying to filter out noisy spans using the filterprocessor + ottl on the collector side. We enabled Boto instrumentation in OpenTelemetry but our flame graph is getting littered with a lot of requests. 99% of the time, it is too much information, but sometimes it is helpful to include a span about the AWS API calls. For example, if a call to S3 takes over a certain amount of seconds, or if there's downtime for an AWS API that affected a call.
Here's an example of noisy spans that I want to be able to filter out:
I tried modifying my Lambda Otel collector config to use this:
But I get an Extension.Crash in the Lambda function with a message like this:
Steps to reproduce if your question is related to an action
See this GitHub repository that I created to demonstrate this issue: https://github.com/kmcquade/otel-lambda-failure-example
What did you expect to see?
Environment
See the example repository mentioned above.
See the example repository mentioned above.