Prometheus time window scraping support

[lynchc]

Rather than having cernan dump all metrics collected in memory on a prometheus scrape, I would like it to present collected metrics for a configurable amount of time.

The goal here is so that we can point multiple prometheus scrapers at the cernan endpoint and duplicate the results for HA

[blt]

@lynchc my thinking is that I'll add a moving window of configurable size to the underlying storage. Metrics will only drop off when they've gotten too old, not when they've been read. @benley does that work for you too?

[benley]

That sounds OK, as long as the metrics are all things that cease to be meaningful if they get too old. If cernan is acting as an event receiver for things that happen like once an hour and get exported as counters or gauges, you probably wouldn't want those to expire.

[blt]

Those would end up expiring, yeah. Say the moving window is 60s and the event is graphite telemetry. Cernan will store that as a an ephemeral, SET and drop it after 60s of receipt. A statsd gauge would persist until it gets reset by some other value.

I guess 'too old' is really a function of input protocol and persistence flags set via that protocol.

[benley]

@brian-brazil suggested that we take a look at how the Prometheus statsd exporter and graphite exporter handle this, because there is a fair bit of functional overlap with cernan there.

[blt]

@benley @brian-brazil got it. I don't read Go all that well so correct me if I'm wrong here.

It looks like what the graphite exporter does is store up samples, notes the timestamp and for some tick interval and expiration age-limit it goes through all the samples and removes the toasted ones:

• https://github.com/prometheus/graphite_exporter/blob/master/main.go#L145

I can't figure out the structure of the statsd example. I could use some help there. Near as I can tell it's creating prometheus values directly on receipt?

• https://github.com/prometheus/statsd_exporter/blob/master/exporter.go#L76

For the prometheus sink it's storing samples

• https://github.com/prometheus/graphite_exporter/blob/master/main.go#L171

and kicking them out via NewConstMetric on... some interval I guess, so long as they aren't old. Seems to me that the graphite samples are just storing the last value to have come in, updating the timestamp for expiration accordingly.

Whenever we aggregate into a Telemetry in cernan we don't update the timestamp because we've figured to this point that the thing on the other end of the sink will care about accurate times and its the sinks responsibility to ensure that Telemetry which get added together are of the "same" time. I guess what I can do in the Prometheus sink is keep track of when the last sample went into the Telemetry aggregate and drop any aggregate Telemetry that are too old.

Now, the other thing here is that cernan does not obey aggregations from input protocols. The statsd_exporter notes it maps like so:

StatsD gauge   -> Prometheus gauge

StatsD counter -> Prometheus counter

StatsD timer   -> Prometheus summary                    <-- indicates timer quantiles
               -> Prometheus counter (suffix `_total`)  <-- indicates total time spent
               -> Prometheus counter (suffix `_count`)  <-- indicates total number of timer events

Since cernan stores everything as quantile structures anyway the feedback I've gotten so far is that it's pretty spiffy just to go ahead and emit that quantile structure into Prometheus. I can change the sink to obey aggregation mapping.

[blt]

This will land as a part of #306.