search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
19.18k stars 1.26k forks source link

why is clickhouse in metrics not able to improve perf in larger group by intervals #2171

Open ankitnayan opened 1 year ago

ankitnayan commented 1 year ago

If we request a chart for a 1hr, the interval between consecutive datapoints is 1m. If we change the chart range to 1 month with interval of 1hr, the aggregation becomes faster in prometheus.

Why is there such a difference between Prometheus and our queries in clickhouse? How to bridge the gap in perf due to that.

srikanthccv commented 1 year ago

Prometheus knows how to look up the chunk file where the data is stored for the timestamp that it should show/return value and reads the data from it. We, however, use toStartOfInerval to normalise the raw timestamp and take the last value while still reading the whole data. The more data we read, the slower our queries run.