Open mxiamxia opened 2 months ago
Added this component to the Collector SIG agenda as a vendor proposed component, the next maintainer on the rotating sponsor list should pick this up. cc @crobert-1
Thank you! Missed the collector SIG discussions for this week. Will join the discussions next week and provide more details about the proposal.
High Cardinality Metrics Protection which helps users to cap the total number of unique metrics for their services before sending it to the destination.
Could we consider achieving this in a non vendor specific way?
AWS Platform related telemetry attributes enrichment
Can we detail this?
Will keep design updated in https://docs.google.com/document/d/1YybdUN2__QL7mSF86ZBCN7AL4W5eXDgyoVzEGCpwz58/edit?usp=sharing
Thx @mxiamxia I think it would be beneficial to detail what is AWS specific in the 4 components of this processor, and why a combination of the existing processors cannot achieve the same results
@bryan-aguilar We had discussed this in a couple collector sig meetings, are you able to sponsor this?
High Cardinality Metrics Protection which helps users to cap the total number of unique metrics for their services before sending it to the destination.
Could we consider achieving this in a non vendor specific way?
Hi @jeromeinsf , sorry for the late response. The MetricLimiter coming with this component primarily having 2 functions - 1) group the metrics on a list of specific metric attributes, then count and sort the occurrences of each grouped metrics using Count-Min Sketch(CMS). 2) take the actions to the metrics having less occurrences found in CMS when the cardinality threshold limit is met. Currently, these 2 piece of functions are implemented in a very specific way based on the customer experience designed for AppSignals. With extra efforts, I think it is possible to abstract some pieces into general purpose. With the current proposal, we probably want to defer this efforts and it requires a follow up with community for the further discussion on the abstractions.
Thx @mxiamxia I think it would be beneficial to detail what is AWS specific in the 4 components of this processor, and why a combination of the existing processors cannot achieve the same results
Thx @mxiamxia I think it would be beneficial to detail what is AWS specific in the 4 components of this processor, and why a combination of the existing processors cannot achieve the same results
Regarding 4 components listed, AppSignals users can leverage a list of existing processors including attributesprocessor
, spanprocessor
, k8sprocessor
and spanprocessor
and then configure it in certain way for processing AppSignals data to fulfill partial of requirements for ApplicationSignals, and some new things introduced like EKS/K8s Pod IP resolver, MetricLimiter will still be needed by introducing this new component. So we build on inclusive vendor specific component to achieve all these requirement so users won't need to worry about any configuration things.
Hey @mxiamxia,
Do you mind organising a sponsor for this component so that subsequent PRs can flow through.
We will need to find a new Sponsor. @bryan-aguilar from AWS is not available anymore to help sponsor this. We are fine with getting a new sponsor not from AWS. We will help ensure that PRs for this new processor will still be reviewed/owned from AWS representatives with commitment.
Hi @crobert-1, would you be free to be the sponsor for this component? As mentioned earlier, @bryan-aguilar isn't available to sponsor this.
As this component is vendor-specific for AWS ApplicationSignals, @mxiamxia, @bjrara, and I will help with reviewing/supporting the contributions to this component as representatives.
Hello @jj22ee, I recently sponsored a different vendor-specific component, so I've moved to the back of the rotating sponsors list.
I see a few different contributors here proposing to work on this, are any of you OpenTelemetry project members? I ask because if the contributor is a member we'll automatically assign a sponsor. Otherwise, you may need to find a sponsor, or become an OpenTelemetry member. Here are the guidelines for contributing new vendor-specific components.
@mxiamxia is a member of open-telemetry (checking from - https://github.com/orgs/open-telemetry/people). In this case, it sounds like we'll need an automatically assigned sponsor.
As discussed in a Collector SIG meeting, @djaglowski could be the sponsor for this component. Much appreciated!!
Why isn't this being added to the ADOT collector?
This component is not for ADOT only. The reason to add this component into upstream, is so that existing consumers of any OTel collector (non-ADOT) will be able to use ApplicationSignals with their current OTel setup.
This component is not for ADOT only. The reason to add this component into upstream, is so that existing consumers of any OTel collector (non-ADOT) will be able to use ApplicationSignals with their current OTel setup.
Generally, hosting a component upstream isn't necessary in order to allow others to pull it into any other OTel collector. As long as the component's go module is publicly accessible it should be possible. If this is the only reason then is it worth having the community take this on as an obligation?
The MetricLimiter coming with this component primarily having 2 functions - 1) group the metrics on a list of specific metric attributes, then count and sort the occurrences of each grouped metrics using Count-Min Sketch(CMS). 2) take the actions to the metrics having less occurrences found in CMS when the cardinality threshold limit is met. Currently, these 2 piece of functions are implemented in a very specific way based on the customer experience designed for AppSignals. With extra efforts, I think it is possible to abstract some pieces into general purpose. With the current proposal, we probably want to defer this efforts and it requires a follow up with community for the further discussion on the abstractions. ... Regarding 4 components listed, AppSignals users can leverage a list of existing processors including attributesprocessor, spanprocessor, k8sprocessor and spanprocessor and then configure it in certain way for processing AppSignals data to fulfill partial of requirements for ApplicationSignals, and some new things introduced like EKS/K8s Pod IP resolver, MetricLimiter will still be needed by introducing this new component. So we build on inclusive vendor specific component to achieve all these requirement so users won't need to worry about any configuration things.
I'm not sold on the idea that there's anything vendor-specific here other than configuration settings. Is it fair to say that this proposal could be separated into two parts?
The purpose and use-cases of the new component
Amazon CloudWatch ApplicationSignals utilizes the OTel Auto-instrumentation SDKs to automatically instrument applications running on AWS, and generates the custom application metrics, traces and log to monitor the application health and track long-term application performance. Currently, the generated telemetry data are processed by CloudWatch Agent before being sent to the AWS backend.
This proposal is to contribute ApplicationSignals components in CloudWatch Agent to OpenTelemetry Collector community.
The main functionalities
Example configuration for the component
Telemetry data types supported
Is this a vendor-specific component?
Code Owner(s)
mxiamxia@; bjrara@
Sponsor (optional)
Additional context
The AwsApplicationProcessor provides a data-processing pipeline which has 4 major components:
No response