Singleton Receiver Creator

skhalash commented 1 month ago

The purpose and use-cases of the new component

A receiver creator that can wrap an arbitrary sub-receiver and ensure that only one instance of this sub-receiver is active at a time in a high-availability OTel Collector setup. This setup is useful in a situation where there are multiple collector replicas running, but only one of them is producing telemetry (metrics, data, logs) at a time. The rest of the replicas are not active (standby mode). This mechanism is implemented using leader election.

Example configuration for the component

receivers:
  singleton_receiver_creator:
    auth_type: serviceAccount
    leader_election:
      lease_name: foo
      lease_namespace: bar
      lease_duration: 15s
      renew_deadline: 10s
      retry_period: 2s
    receiver:
      otlp:
        protocols:
          grpc:
            endpoint: ""

Telemetry data types supported

traces, metrics, and logs

Is this a vendor-specific component?

[ ] This is a vendor-specific component
[ ] If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
[ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@skhalash @a-thaler

Sponsor (optional)

No response

Additional context

I work at SAP on a project called Kyma: https://kyma-project.io/#/. In Kyma, we recently developed such a receiver and we are already testing it out in a production setup: https://github.com/kyma-project/opentelemetry-collector-components/tree/main/receiver/singletonreceivercreator.

We’ve noticed discussions in the community about introducing leader election in the k8sobjects and k8scluster receivers. That's why we believe that a generic mechanism could be quite beneficial. If there’s interest, we are ready to contribute it to the Open Telemetry project.

ChrsMark commented 1 month ago

That sounds interesting. A similar approach is used in Metricbeat for covering the cluster level metrics collection when running as part of a Daemonset.

II assume we want to achieve something similar here where the Collector will be running only as a Daemonset and the leader will be responsible for enabling the k8sclusterreceiver and k8sobjectsreceiver?

While this is useful from user experience perspective, from my past experience, it can be problematic when it comes to scale. When you deploy the Collector as a Daemonset you set the resource limits according to the Pod's needs. However one of the Pods will be the leader and hence would require extra resources. In order to support this you need to increase the resource requests/limits for the whole Daemonset, but not all of the Pods will actually need those resources. This can be confusing. In addition, such a feature might also affect the load that the Collectors put to the K8s API. I have seen such issues in the past but I don't have something specific to share here. So in summary, such a feature would need to explicitly document the pros/cons etc (and maybe be tested accordingly) and properly set the expectations for the users.

skhalash commented 4 weeks ago

Hey @ChrsMark! Thanks for the feedback. Yes, exactly the leader is responsible for enabling a sub-receiver (such as k8sclusterreceiver and k8sobjectsreceiver) if the collector is running as a DaemonSet or a Deployment.

Yes, I fully agree with the concerns about resource limits/requests. However, as you said, it should be properly documented. Not sure if there is a way to workaround it.

Regarding putting some extra load on the k8s API - do you mean querying/updating leases?

ChrsMark commented 1 day ago

Regarding putting some extra load on the k8s API - do you mean querying/updating leases?

@skhalash yes, but this is something that can also be properly documented along with some perf tests results so as users be aware of any possible impact to their clusters.

open-telemetry / opentelemetry-collector-contrib