[PIP-214][broker]Add broker level metrics statistics and expose to prometheus

yyj8 commented 1 year ago

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation

Currently, pulsar does not statistics broker level metrics, and all relevant metrics are 0 by default.

When the number of topic partitions reaches more than 100000 or even millions, and the topic level metrics are exposed (exposeTopicLevelMetricsInPrometheus=true), if you want to query the metrics of the broker dimension, you need to summarize all topics, and the performance becomes very poor, or even the results cannot be queried from the promethus. Common broker level metrics include:

pulsar_topics_count
pulsar_subscriptions_count
pulsar_producers_count
pulsar_consumers_count
pulsar_rate_in
pulsar_rate_out
pulsar_throughput_in
pulsar_throughput_out
pulsar_storage_size
pulsar_storage_logical_size
pulsar_storage_write_rate
pulsar_storage_read_rate
pulsar_msg_backlog

We need to statistics the metrics of the broker dimension and expose them to prometheus to improve the performance of the monitoring of the broker dimension. Modify the original metrics name as follows:

pulsar_broker_topics_count
pulsar_broker_subscriptions_count
pulsar_broker_producers_count
pulsar_broker_consumers_count
pulsar_broker_rate_in
pulsar_broker_rate_out
pulsar_broker_throughput_in
pulsar_broker_throughput_out
pulsar_broker_storage_size
pulsar_broker_storage_logical_size
pulsar_broker_storage_write_rate
pulsar_broker_storage_read_rate
pulsar_broker_msg_backlog

Solution

No response

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

[X] I'm willing to submit a PR! PR:https://github.com/apache/pulsar/pull/18116

asafm commented 1 year ago

I'm not sure a PIP is formatted as you did. It has a certain process to it.

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.

poorbarcode commented 1 year ago

Hi @yyj8

What is the status of this PIP? I see https://github.com/apache/pulsar/pull/19047 is merged. So my understanding is that all things are completed, right? If yes, Could we send an email to close this PIP?

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.

apache / pulsar