[improve][broker] Enhance broker metrics to support URL parameter filtering

[zwOvO]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

The motivation for enhancing broker metrics to support URL parameter filtering in Pulsar is to provide users with a more flexible and convenient way to retrieve specific metrics. By allowing filtering through URL parameters, users can easily specify the metrics they are interested in, making it easier to monitor and analyze the performance of the Pulsar broker. This enhancement would improve the usability and accessibility of the metrics system, ultimately leading to a better user experience for Pulsar users.

Solution

for example:

1. /metrics?m=jvm_.*
2. /metrics?m=pulsar_managedLedger_client_per_channel_bookie_clientbookie.*

Alternatives

No response

Anything else?

Currently, the broker metrics should be enhanced to allow filtering by URL parameters. This enhancement will enable users to filter metrics using URL parameters $ curl --location --request GET 'http://127.0.0.1:8080/metrics' 2>/dev/null|wc -l 100000

$ curl --location --request GET 'http://127.0.0.1:8080/metrics' 2>/dev/null|grep 'jvm_'|wc -l 20

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!

[Georift]

I'd really appreciate this change, just last week we were running into cases where we'd have loved to filter the metrics returned.

Do you think we would we be able to skip background work collecting metrics that were not matched by the query parameters? In some cases our metrics scraping can cause significant load on the brokers, perhaps this could be a mitigation?

[avirlrma]

Hey everyone, I would like to tackle this one if that's okay with the team.

[avirlrma]

I was able to come up with two potential approaches for implementing this feature:

Approach 1: Modify Individual Metric Generation Methods

This approach involves modifying each metric generation method to include filtering logic.

Pros:

• Direct and straightforward implementation
• Potential for better performance as filtering happens at the source
• Allows for fine-grained control over filtering for each metric type

Cons:

• Requires changes to multiple classes and methods (NamespaceStatsAggregator, PrometheusMetricsProvider, TransactionAggregator, MetricsProviders etc.)
• May lead to code duplication
• More difficult to maintain and update in the future

Approach 2: Subclass SimpleTextOutputStream

This approach involves creating a new FilteringTextOutputStream that extends SimpleTextOutputStream and incorporates the filtering logic.

Pros:

• Centralized filtering logic
• Minimal changes to existing metric generation methods
• Easier to maintain and extend in the future
• Consistent filtering across all metric types

Cons:

• Slight performance overhead due to buffering each line
• May use more memory temporarily while buffering lines
• Less fine-grained control over filtering for specific metric types

Implementation Details (Common to Both Approaches)

1. URL Parameter:

   • We will use the query parameter "m" to specify the filter.
   • Example: /metrics?m=jvm_.*

2. Filter Format:

   • The filter will use Java regular expressions for maximum flexibility.

   Having suggested both the approaches, I am leaning towards approach#1, as it more of a long-term solution to metric filtering problem, just not when output stream is involved.

   Would appreciate feedback on this. Thanks! CC: @zwOvO