Enable autoscaling otel-collectors

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool

https://signoz.io

Other

19.53k stars 1.32k forks source link

Enable autoscaling otel-collectors #2059

Open ankitnayan opened 1 year ago

ankitnayan commented 1 year ago

Test and enable autoscaling of otel-collectors as ingestion load increases.

which metric to base auto-scaling logic on? CPU usage of otel-collectors or data being dropped or exporter queue getting filled?
verify load is being distributed amongst otel-collectors as there are new replicas added to handle the ingestion load

ankitnayan commented 1 year ago

Autoscaling when sum of received datapoints in metrics/traces/logs > 10K/s might be a good starting point.

ankitnayan commented 1 year ago

Autoscaling when sum of received datapoints in metrics/traces/logs > 10K/s might be a good starting point.

@prashant-shahi please try this out first

prashant-shahi commented 1 year ago

Chart PR: https://github.com/SigNoz/charts/pull/161