search

nerdswords / yet-another-cloudwatch-exporter

Prometheus exporter for AWS CloudWatch - Discovers services through AWS tags, gets CloudWatch metrics data and provides them as Prometheus metrics with AWS tags as labels
Apache License 2.0
953 stars 328 forks source link

[FEATURE] Ensure info metrics will have compatibility with prometheus OTEL implementation #1436

Open kgeckhart opened 2 months ago

kgeckhart commented 2 months ago

Is there an existing issue for this?

Feature description

YACE exports info for resource metadata like tags. This normalizes the data off of the CloudWatch metrics and produces a simplified metric model that can be built up via prometheus joins.

The problem is that YACE chose to export these info metrics with a value of 0. This goes against the accepted prometheus implementation for info metrics where the value exported is 1, https://www.robustperception.io/why-info-style-metrics-have-a-value-of-1/. So far this hasn't really been a big deal other than joins need to use a + operator vs the a * to avoid breaking values.

But now that prometheus is moving to be more otel compatible they are planning to implement simplified query support for info metrics, https://github.com/prometheus/prometheus/issues/12629. The problem is that whatever is done with prometheus is likely to be incompatible with YACE as YACE does not follow the accepted standard.

We should find a way to safely migrate from exporting a 0 value from info metrics to a 1 to ensure we reap the benefits of the upcoming info metric work.

What might the configuration look like?

Not sure

Anything else?

If we wait long enough this is more than likely to be considered a bug rather than a feature.

Pokom commented 1 month ago

I actually just ran into this trying to join data from https://github.com/grafana/cloudcost-exporter/blob/main/docs/metrics/aws/ec2.md to calculate the costs of an instance and was surprised to see the value is returned as zero. Is there any specific reason why info metrics are exported as zero? IE will it be a backwards incompatible change?

Appreciate that you dropped a breadcrumb on how to overcome it using the + operator. I've usually used a count or group hack to get the value back to one.

kgeckhart commented 1 month ago

Is there any specific reason why info metrics are exported as zero? IE will it be a backwards incompatible change?

I'm not sure as I think this one largely predates me and @cristiangreco. I found that it's been 0 for at least 6 years now, https://github.com/nerdswords/yet-another-cloudwatch-exporter/pull/4/files#diff-dec443ff93e42df03c5316555338fccf81a2cb1ec4a646276a02cf034e6895cdR88-R127. Pretty sure it's been zero since the info metrics existed.

Changing it to 1 now is going to break anyone who currently has queries which depend on the value being zero 😬