SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
We are using distributed ClickHouse within Signoz, with a 700GB PVC attached. Despite configuring log retention to 1 hour, and traces and metrics retention to 1 day, we are encountering persistent storage full issues.
Database Sizes:
signoz_traces: 636.73 GB
signoz_metrics: 10.47 GB
signoz_logs: 2.03 GB
system: 0.07 GB
Largest Tables in signoz_traces:
signoz_index_v2: 486.07 GB
span_attributes: 14.47 GB
durationSort: 94.40 GB
signoz_spans: 39.92 GB
Observation:
The signoz_index_v2 table is consuming a significant portion of storage, holding data beyond the configured retention period (1 day).
Query Timestamp Anomaly:
Despite a configured retention of 1 day, data with timestamps up to 2024-07-09 (2 days ago) persists within signoz_index_v2.
Request:
Please investigate why data retention policies are not being applied as expected and provide guidance on resolving this storage issue.
We are using distributed ClickHouse within Signoz, with a 700GB PVC attached. Despite configuring log retention to 1 hour, and traces and metrics retention to 1 day, we are encountering persistent storage full issues.
Database Sizes: signoz_traces: 636.73 GB signoz_metrics: 10.47 GB signoz_logs: 2.03 GB system: 0.07 GB Largest Tables in signoz_traces: signoz_index_v2: 486.07 GB span_attributes: 14.47 GB durationSort: 94.40 GB signoz_spans: 39.92 GB Observation: The signoz_index_v2 table is consuming a significant portion of storage, holding data beyond the configured retention period (1 day).
Query Timestamp Anomaly: Despite a configured retention of 1 day, data with timestamps up to 2024-07-09 (2 days ago) persists within signoz_index_v2.
Request: Please investigate why data retention policies are not being applied as expected and provide guidance on resolving this storage issue.