grafana / k6

A modern load testing tool, using Go and JavaScript - https://k6.io
GNU Affero General Public License v3.0
25.77k stars 1.27k forks source link

Cumulative gauge/counter metric #1340

Open na-- opened 4 years ago

na-- commented 4 years ago

Following the discussion in https://community.k6.io/t/is-it-possible-to-create-bespoke-metrics-to-track-system-state/446, I think a new metric deserves consideration.

We should do some research on how other metrics libraries handle this, and as I said in the community forum, there are workarounds, but I can see the value of having something like a cumulative gauge metric. It will behave like a gauge/counter, keeping only the last value, but adding values to it won't replace the previous value, rather it will add to it, like a Counter. However, most importantly, when it's emitted to external outputs, k6 will emit the new cumulative value, not just the delta, very much unlike a Counter.

For example, if I run this script with k6 run --out json:

import { Gauge } from 'k6/metrics';

var newMetric = new CumulativeGauge('cumulative_guage');

export default function () {
    newMetric.add(3);
    newMetric.add(1);
    newMetric.add(-3);
    newMetric.add(5);
}

we should get something like this:

{"type":"Point","data":{"time":"...","value":3,"tags":{"group":""}},"metric":"cumulative_guage"}
{"type":"Point","data":{"time":"...","value":4,"tags":{"group":""}},"metric":"cumulative_guage"}
{"type":"Point","data":{"time":"...","value":1,"tags":{"group":""}},"metric":"cumulative_guage"}
{"type":"Point","data":{"time":"...","value":6,"tags":{"group":""}},"metric":"cumulative_guage"}

and the cumulative_guage value in the end-of-test summary should be 6

na-- commented 4 years ago

Because this "new" metric will behave exactly like a Counter in regards to things like threshold calculations and the end-of-test summary, it might make sense to just add a flag (or a new constructor) to the Counter metric (similar to the isTime flag), so k6 just emits them in a cumulative fashion to external outputs.

Also, given the current real-time metrics architecture of k6, either option is going to be somewhat complicated to implement. We currently output metrics to external outputs before k6 itself has processed them internally, so we'd need to seriously refactor things if we ever want this (i.e. https://github.com/loadimpact/k6/issues/1075 becomes even more complicated).

mstoykov commented 4 years ago

Hm... how would this work with distributed execution ?

na-- commented 4 years ago

Exactly in the same way that a Counter would - you can sum the values from 2 instances to get the total end value.

mstoykov commented 8 months ago

This aligns with the definition of cumulative Sums from otels' spec. Whcih we will need to match with our current Counters it either way ... it seems.

Relevant for #2557