JSON output format doesn't include thresholds

[fogone]

Version

> k6 version                                                                                     
k6 v0.24.0

Test

import { Rate } from "k6/metrics";

const rate = new Rate("some-rate");

export const options = {
    iterations: 1,
    thresholds: {
        "some-rate": ["rate>0.9"]
    }
};

export default function () {
    rate.add(true);
};

Run

 k6 run --out json=result.json  tmp.js

          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /    
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     output: json=result.json
     script: tmp.js

    duration: -,  iterations: 1
         vus: 1, max: 1

    done [==========================================================] 1 / 1

    data_received........: 0 B     0 B/s
    data_sent............: 0 B     0 B/s
    iteration_duration...: avg=61.01µs min=61.01µs med=61.01µs max=61.01µs p(90)=61.01µs p(95)=61.01µs
    iterations...........: 1       0/s
  ✓ some-rate............: 100.00% ✓ 1   ✗ 0  
    vus..................: 1       min=1 max=1
    vus_max..............: 1       min=1 max=1

Json

{"type":"Metric","data":{"name":"some-rate","type":"rate","contains":"default","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"some-rate"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.233005026+03:00","value":1,"tags":{"group":""}},"metric":"some-rate"}
{"type":"Metric","data":{"name":"data_sent","type":"counter","contains":"data","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"data_sent"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.233007615+03:00","value":0,"tags":{"group":""}},"metric":"data_sent"}
{"type":"Metric","data":{"name":"data_received","type":"counter","contains":"data","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"data_received"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.233007615+03:00","value":0,"tags":{"group":""}},"metric":"data_received"}
{"type":"Metric","data":{"name":"iteration_duration","type":"trend","contains":"time","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"iteration_duration"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.233007615+03:00","value":0.061017,"tags":{"group":""}},"metric":"iteration_duration"}
{"type":"Metric","data":{"name":"iterations","type":"counter","contains":"default","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"iterations"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.233012443+03:00","value":1,"tags":null},"metric":"iterations"}
{"type":"Metric","data":{"name":"vus","type":"gauge","contains":"default","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"vus"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.269144407+03:00","value":1,"tags":null},"metric":"vus"}
{"type":"Metric","data":{"name":"vus_max","type":"gauge","contains":"default","tainted":null,"thresholds":[],"submetrics":null,"sub":{"name":"","parent":"","suffix":"","tags":null}},"metric":"vus_max"}
{"type":"Point","data":{"time":"2019-06-14T17:42:54.269144407+03:00","value":1,"tags":null},"metric":"vus_max"}

Let's look at our some-metric line from prev file:

{
  "type": "Metric",
  "data": {
    "name": "some-rate",
    "type": "rate",
    "contains": "default",
    "tainted": null,
    "thresholds": [], // <-- ¯\_(ツ)_/¯ 
    "submetrics": null,
    "sub": {
      "name": "",
      "parent": "",
      "suffix": "",
      "tags": null
    }
  },
  "metric": "some-rate"
}

So, as you can see, there is no threshold name in the "thresholds" array, as it described in the spec. I can't know that some thresholds was defined. It's really big problem when I need to analyze results, because I don't know were all thresholds passed or not.

[na--]

Thanks for reporting this! We'd have to do a bit of code archeology to understand why the thresholds were emitted in the metrics and why they aren't anymore, but this seems like a very old regression... Did a quick test and k6 v0.20 lists the thresholds, but k6 v0.21.0 and subsequent version do not :confused:

And just to be clear, since (as far as I remember) k6 emits the "type": "Metric" JSON values the first time a metric is emitted, you are only ever going to see a list with the defined thresholds for that metric, but it won't show whether the threshold passed or not. To show if a threshold is crossed or not would be a separate feature/enhancement task, more in line with https://github.com/loadimpact/k6/issues/355 or https://github.com/loadimpact/k6/issues/351.

[lightindigo]

I see, there's a regression on version 0.48/0.49

[codebien]

Hey @lightindigo, can you open a bug report issue, please?