Closed cykl closed 1 month ago
Should the default relabel configuration be updated to take those metrics into account or am I doing something wrong?
Should definitely be added. The current relabeling rules are based on the documented metrics, but clearly this isn't enough to get everything done correctly.
What Cassandra version was used?
In any case, the metrics names come from these lines:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeInboundMetrics.java#L54 https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L122
So we will need a relabeling rule that would catch these for IPv4 & IPv6 and put the target node as a label instead of allowing it to stay in the metricName.
We are using Cassandra 4.1.3 .
Where are the rules defined? There is this file https://github.com/k8ssandra/management-api-for-apache-cassandra/blob/f85032fa787a4e2a42a37ad16a94eeefa70ff907/management-api-agent-common/src/test/resources/collector-full.yaml (linked from the doc). Butsrc/test/resources
leads me to believe that it's not the file that's being used.
The default rules (you can define your own and they get appended to these) are in https://github.com/k8ssandra/management-api-for-apache-cassandra/blob/master/management-api-agent-common/src/main/resources/default-metric-settings.yaml
And here's an example how to append your own rules (like the ones I added to the default):
CassandraDatacenter changes:
And the mounted ConfigMap:
While inspecting Cassandra related metrics in our Prometheus server, I noticed that some metrics were badly re-labelled from the Cassandra model to the Prometheus model leading to numerous Prometheus metric .
For example:
I'm using the new metric endpoint without any additional configuration.
Impacted metrics are the following (I stripped the port, ip address, etc. to obtain expected metric name):
None of those metrics seems documented in https://cassandra.apache.org/doc/stable/cassandra/operating/metrics.html#client-metrics but it's easy to find them in Cassandra source code.
Should the default relabel configuration be updated to take those metrics into account or am I doing something wrong?