search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.8k stars 563 forks source link

Port non-persistent metrics to label guarded one #12805

Open wenym1 opened 11 months ago

wenym1 commented 11 months ago

Is your feature request related to a problem? Please describe.

Potential solution to https://github.com/risingwavelabs/risingwave/issues/6855

Currently we have many metrics with labels bind to non-persistent identities, such executors, sinks or tables. Most of these metrics are not dropped when corresponding identities are dropped, and the metrics are leaked and lost track, and may get piled up when the instance is long running and cause OOM eventually.

For some metrics, they currently solve the problem by collecting the labels and later explicitly call remove_label_values on the labels, which is verbose and not general enough for other cases to use.

Describe the solution you'd like

In https://github.com/risingwavelabs/risingwave/pull/12653, we implemented a label guarded metrics, which stores the label when the metric variable is created with with_label_values, and automatically call remove_label_values when the metrics is dropped.

In this way, we only need to change the type of the metrics field from the original one (e.g IntCounter) to the label guarded one (e.g. LabelGuardedIntCounter), and register the metrics with a new macro with the same macro parameters as the original one, and the code of metrics report can remain the same.

Describe alternatives you've considered

No response

Additional context

No response

BugenZhao commented 11 months ago

+1 for this. Previous similar idea: https://github.com/risingwavelabs/risingwave/pull/9756#issuecomment-1552777364

hzxa21 commented 10 months ago

@wenym1 Feel free to reassign.