search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
18.1k stars 1.15k forks source link

Issue: Persistent Volume Claim (PVC) Storage Full in Signoz with Distributed ClickHouse #5467

Closed anshulwalia128 closed 1 month ago

anshulwalia128 commented 1 month ago

We are using distributed ClickHouse within Signoz, with a 700GB PVC attached. Despite configuring log retention to 1 hour, and traces and metrics retention to 1 day, we are encountering persistent storage full issues.

Database Sizes: signoz_traces: 636.73 GB signoz_metrics: 10.47 GB signoz_logs: 2.03 GB system: 0.07 GB Largest Tables in signoz_traces: signoz_index_v2: 486.07 GB span_attributes: 14.47 GB durationSort: 94.40 GB signoz_spans: 39.92 GB Observation: The signoz_index_v2 table is consuming a significant portion of storage, holding data beyond the configured retention period (1 day).

Query Timestamp Anomaly: Despite a configured retention of 1 day, data with timestamps up to 2024-07-09 (2 days ago) persists within signoz_index_v2.

Request: Please investigate why data retention policies are not being applied as expected and provide guidance on resolving this storage issue.

welcome[bot] commented 1 month ago

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.