To push operator metering metrics to Kafka or NoSQL DBs

kube-reporting / metering-operator

The Metering Operator is responsible for collecting metrics and other information about what's happening in a Kubernetes cluster, and providing a way to create reports on the collected data.

Apache License 2.0

339 stars 90 forks source link

To push operator metering metrics to Kafka or NoSQL DBs #1150

Open amalkurup89 opened 4 years ago

amalkurup89 commented 4 years ago

Description :

From the current documentation & whatever experimentation I have done on operator metering, I don't see a feature to upload or push the operator metering metrics to a NoSQL database like Cassandra or Kafka topic instead of Presto DB

This can be useful to query operator metering metrics from web applications as Presto is relatively slow

Please share your thoughts on this or are you folks looking at any such feature soon

Motivation :

Feature enhancement for operator metering

bentito commented 4 years ago

Hi Amal, I don't think we've been looking to cut Presto out of the loop. We actually now have a connector we are contributing to the Presto project that allows Presto to directly query Prometheus (https://github.com/prestosql/presto/pull/2321). As we transition the metering code base to use Presto queries rather than the roundabout manner of Prometheus to Hive to Presto, I think the logical streamlining will also result in a speed up as well. -Brett

timflannagan commented 4 years ago

On top of what Brett mentioned, if you're looking to speed up Presto query execution, a more immediate fix would be to scale some Presto "worker" replicas or pump up the CPU/memory limits for that Pod: https://github.com/operator-framework/operator-metering/blob/master/manifests/metering-config/recommended-resource-limits.yaml#L38-L63