search

aws-observability / aws-otel-lambda

AWS Distro for OpenTelemetry - AWS Lambda
https://aws-otel.github.io/
Apache License 2.0
148 stars 56 forks source link

Decouple processor needs to be added #842

Open mcaparnacoxauto opened 9 months ago

mcaparnacoxauto commented 9 months ago

Is your feature request related to a problem? Please describe.

We are currently exploring the possibility of utilizing the Decouple Processor, however, we've noticed that no processors are available as per the information provided on this link: aws-otel-lambda. This presents a challenge as the Decouple Processor offers unique features that could significantly enhance the performance of our Lambda layer.

Describe the solution you'd like

The Decouple Processor is specifically engineered to circumvent the limitations inherent in Lambda. It does this by reducing overhead and focusing on aspects that are directly relevant to Lambda, making it a more suitable choice compared to other processors. Therefore, it would be highly beneficial to include it in our setup.

One of the key advantages of the Decouple Processor is its awareness of the Lambda lifecycle. This means it can prevent the environment from being frozen or shut down until all pending traces/metrics/logs have been successfully exported. This feature alone could significantly boost the performance of the otel layer.

Furthermore, our Lambda is heavily utilized, processing millions of transactions. Without the Decouple Processor, these transactions could potentially slow down in the ADOT version, affecting overall performance.

Describe alternatives you've considered

We have considered switching to opentelemetry-lambda, but this alternative does not include the AWS services that we require.

In conclusion, the inclusion of the Decouple Processor in our setup could provide significant performance improvements, especially considering the high volume of transactions our Lambda processes. We believe this feature request is worth considering to enhance the efficiency and effectiveness of our operations.

lukep-coxauto commented 8 months ago

Hello, could we please get some comments by AWS on this issue? It is blocking us from using this layer because the processor is not in this layer, despite being added to the underlying opentelemetry-lambda framework that this layer is built on.

As an alternative, could someone provide clarification on the differences between using this ADOT OTEL layer and using opentelemetry-lambda directly?

vasireddy99 commented 7 months ago

Hello, ADOT Lambda Layer consists a subset of components from the upstream and we sometimes apply patches(backport) as needed for fixes to avoid breaking changes etc., more information on ADOT Lambda here

lukep-coxauto commented 7 months ago

Hi @vasireddy99 thank you for the response!

Would you mind providing more direct answers to clarify the AWS team's stance on the following?

  1. Will the decouple processor ever be included in the aws-otel-lambda project, or is this out of scope for this project?

  2. Is there some type of comparison between using aws-otel-lambda and opentelemetry-lambda? I cannot tell if one is a superset of the other, or if they are divergent implementations. the README for this project says

    As a downstream Repo of opentelemetry-lambda

which indicates to me that this repo is a derived repo based on the previous one.

  1. if we choose to use opentelemetry-lambda directly, are we losing any integration with Amazon functionality that the aws-otel-lambda layer provides? another way to phrase this would be, what is the reason to choose this project over the other one? the README states

    aws-otel-lambda publishes AWS managed OpenTelemetry Lambda layers that are preconfigured for use with AWS services and bundle the reduced ADOT Collector. Users can onboard to OpenTelemetry in their existing Lambda functions by adding these ready-made layers directly.

but what does that mean for me? do I want the reduced collector? what preconfiguration with AWS services were done?

vasireddy99 commented 7 months ago

Hi Luke,

Will the decouple processor ever be included in the aws-otel-lambda project, or is this out of scope for this project?

We don't enough info about the timeline on when it will be included, cc: @mhausenblas

Is there some type of comparison between using aws-otel-lambda and opentelemetry-lambda? I cannot tell if one is a superset of the other, or if they are divergent implementations. the README for this project says

which indicates to me that this repo is a derived repo based on the previous one.

ADOT Lambda Layer consists a subset of components from the upstream and we sometimes apply patches(backport) as needed for fixes to avoid breaking changes etc., more information on ADOT Lambda here

yes we use opentelemetry-lambda repo as a git sub module

if we choose to use opentelemetry-lambda directly, are we losing any integration with Amazon functionality that the aws-otel-lambda layer provides?

For the java lambda layers, we made a patch to avoid this breaking change, other than that for the other layers, no integration functionality is lost in general when using otel layers.

if you are using the SDK layer with in-process exporter flushing the telemetry to the destination, reduced collector may not be necessary.

mhausenblas commented 7 months ago

ADOT PM here. Thanks @vasireddy99 for providing some answers already and to add to this thread:

Will the decouple processor ever be included in the aws-otel-lambda project, or is this out of scope for this project?

At the current point in time it's not on our roadmap, but we're always looking for feedback (such as found in this convo) which then influences our prioritization.

if we choose to use opentelemetry-lambda directly, are we losing any integration with Amazon functionality that the aws-otel-lambda layer provides?

Technical or feature considerations aside: the same holds for the Lambda layer as is true for other integrations such as the EKS add-on (using the upstream OpenTelemetry Operator internally): when using ADOT and you're on AWS Enterprise Support , then you can get direct support from AWS, whereas if you're using upstream directly (such as layers here in this case or the OpenTelemetry Operator in the Kubernetes context) then the support you can expect is mainly: 1. community based, and 2. on a best-effort basis (by ourselves) here via GitHub.

Dr-Emann commented 6 months ago

I believe related to #886 and #787

cgilling commented 6 months ago

I just want to add my voice to wanting to get this processor added, seems like it shouldn't be a lot of work to add it since its already a part of the collector. I'm stuck with not really being able to use this since I don't feel comfortable with the added latency that not having the decouple processor adds to user requests, but at the same time I can't use the awsxray trace support with the layers provided by the upstream repo.

bhaskarbanerjee commented 4 months ago

@mhausenblas @vasireddy99 Please refer to this documentation. It says AWS's managed OpenTelemetry Lambda Layer utilizes OpenTelemetry Lambda Layer to export telemetry data asynchronously from AWS Lambda. As far as I understand, ADOT layer is a wrapper on top of open telemetry, which is synchronous in nature as documented here

The only way to make it async is by including decoupled processor which you are saying is still not part of the ADOT layer, nor is there a roadmap so far.

In that case, I am failing to understand why aws doc is claiming that the process is async.

mhausenblas commented 4 months ago

@bhaskarbanerjee thanks for the question and will make sure that we fix the (docs) confusion.

github-actions[bot] commented 4 weeks ago

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

fitz-vivodyne commented 2 weeks ago

still relevant

fitz-vivodyne commented 2 weeks ago

@mhausenblas can you please confirm if the docs are correct and that OTEL information is exported asynchronously, or if @bhaskarbanerjee 's interpretation is correct and it does in fact block the response to the client.

bhaskarbanerjee commented 1 week ago

@fitz-vivodyne I have confirmed with @mhausenblas that aws exports data synchronously. It seems the documentation is till not updated @mhausenblas

mhausenblas commented 1 week ago

It seems the documentation is till not update

It was, right after we discussed it, see:

https://github.com/aws-observability/observability-best-practices/commit/470cc2be572bf81f9fce21dca670566db9a68de0

bhaskarbanerjee commented 6 days ago

Hey @mhausenblas did that commit get merged and deployed? The page still reads the same.

mhausenblas commented 6 days ago

Hmmm, I see what you mean. Let me investigate.

mhausenblas commented 6 days ago

@bhaskarbanerjee thanks again for bringing this to my attention, it's published.

fitz-vivodyne commented 4 days ago

Thanks for the follow up @mhausenblas. I'm assuming that including the decouple processor is still not on the roadmap (as mentioned here)?

If so I'd like to add a +1, the layer would be a lot more compelling if the decouple processor was included and allowed us to trim some of the excess layer latency.