Open quayly opened 1 year ago
Jira CommentId: 218647 Commented by ekanner:
Just adding an update from the customer around our current proposed workaround, metric override rules:
"Here's my regex for the counter override so far: regex: (.*)_(0rtt$|200$|abandoned$|added$|attempt$|broken$|cancelled$|change$|completed$|created$|destroy$|differs$|directly$|eject$|empty$|error$|errors$|exceeded$|exponential$|evictions$|fail$|failure$|failures$|fallback$|fields$|flood$|flood$|flushes$|frames$|healthy$|headers$|hits$|insertions$|invalid$|latency$|left$|local$|max_size$|merge$|miss$|misses$|mode$|modified$|notify$|ok$|out_of_merge_window$|overflow$|panic$|protos$|pushes$|ratelimited$|reached$|received$|rebuild$|removed$|rejected$|remote$|requests$|reset$|responses$|retry$|rq$|rtt$|sampled$|selected$|small$|stale$|stopped$|stream$|structures$|success$|successes$|terminations$|timeout$|trailers$|triggers$|underscores$|updated$|underscores$|wondow$|zone$) so many different ones and still finding more... so painful"
This bug depends on the open github issue which will make the metadata field carrying metric type information in the Prometheus remote write protocol.
NewRelic Prometheus Configurator Not Assigning Correct Metadata Type By Reading The Metrics File
Description
According to https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-prometheus-agent/migration-guide/#metric-types , any metrics name that doesn't have the suffix total, count, sum, or bucket, it will default to Gauge. We have over 300+ metrics from various application that should be tag as counter instead of gauge.
Expected Behavior
The NewRelic Prometheus Configurator scrapes all of the pods that have the prometheus.io/scrape: "true" and gets the metrics. It should read the metrics file to figure out the correct metedata type "# TYPE" so that it assign the correct tag. This is what nri-prometheus does! However, NewRelic Prometheus Configurator by default that if the metrics suffix is not total, count, sum, or bucket, it will be converted as Gauge.
Please see https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-prometheus-agent/migration-guide/#metric-types for more info.
We have many metrics that doesn't have the suffix name total, count, sum, and buckets in them and they are being marked as Gauge instead of counter. It is causing the data to be incorrect when doing it goes to NewRelic.
We have to manually look thru all the metrics and verify if it the tag is correct or not. If not, we will need to override the metrics type mapping. Instruction on how to update the mapping: https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-remote-write/set-your-prometheus-remote-write-integration/#mapping
Issue with the provided solution are: