grafana / k6

A modern load testing tool, using Go and JavaScript - https://k6.io
GNU Affero General Public License v3.0
24.87k stars 1.23k forks source link

Defining thresholds in `setup` #3424

Open yorugac opened 10 months ago

yorugac commented 10 months ago

Feature Description

Currently thresholds can be defined only in init context, as part of options const object. This has its limitations: if threshold must depend on a value unavailable in init context, it cannot be defined.

Such a value could be:

  1. from Execution API
  2. coming from an extension's functionality

In many cases, one can try to add workarounds: e.g. passing some additional env vars from the outside, etc. But it would be better and have greater readability to have definition of threshold happen in setup instead with something like:

import fancy from 'k6/x/fancy';

export const options = {
  thresholds: {
    ...
  },
};

export function setup() {
  let timeoutSecs = fancy.estimate();
  options.setThreshold({
      'vu_setup': [{
            threshold: `count==${exec.instance.vusInitialized}`,
            abortOnFail: true,
            delayAbortEval: `${timeoutSecs}s`,
       }]
  });
}
...

Suggested Solution (optional)

No response

Already existing or connected issues / PRs (optional)

Somewhat related: https://github.com/grafana/k6/issues/785

codebien commented 8 months ago

If we find a way of defining a good API then probably we don't need to limit this to setup. For sure, it is required for the first iteration, but after https://github.com/grafana/k6/issues/1321 has been implemented then we might be able to set thresholds more dynamically.