Allow plugins to register and report metrics

[xiangfu0]

Current plugins have no way to register metrics(Corresponding MetricsRegistery utils code is in pinot-common), hence no metrics are really published along with each component. We need to implement a way to dynamically extract metrics from plugins, so plugins can report metrics based on their own needs. E.g. filesystem plugins should be able to report the API counter/latency metrics Kafka plugins should be able to report topic partition information and consumer lag etc.

[xiangfu0]

cc: @npawar

[gortiz]

Do we actually need this? Metrics are not that different than logging. We do not have a logging api, we use slf4j. If some dependency we have uses log4j to log, then we can simply use the slf4j bridge to have them.

Going back to metrics, each plugin could include a dependency to dropwizard/micrometer and register their metrics as JMX beans. Then Prometheus collector will see them classified by their tags in the same way that log4j2.xml files can see logs classified by their classes/packages.

In my opinion, with the metrics registry we are adding a bunch of indirections that instead of helping us to have a generic system that is easier to change from one metric library to the other, is making that change more difficult. In fact we are still using yammer when the community is already stopping to use the yammer successor, dropwizard.

The obvious con to have plugins directly using a metric library is that all plugins must use a compatible library version. That could also be said when talking about logging. And that is why mature logging and metric libraries maintain a good backward compatibility.

Another thing to consider is the metrics of third party libraries. Again we can do a comparison with logging. These third party libraries log with slf4j or log4j and we collect their logs with log4j. The same should happen with metrics. They can use mostly whatever they want to register their metrics in the JMX context and we can use Prometheus to collect them. Obviously there are more libraries that register logs than libraries than register metrics, but there are libraries that do that. For example Zookeeper, Helix and Kafka clients expose their own metrics.