grafana / k6

A modern load testing tool, using Go and JavaScript - https://k6.io
GNU Affero General Public License v3.0
26.1k stars 1.27k forks source link

Thresholds using URL may not correlate in browser based tests #4042

Open ankur22 opened 2 weeks ago

ankur22 commented 2 weeks ago

Brief summary

When running a browser test, and navigating to a site (e.g. https://test.k6.io) if the trailing "/" is missing, chrome will add it. All metrics that are associated to that navigation will work with the amended address with the trailing "/" (e.g. https://test.k6.io/). This can be problematic when working with thresholds, since the user may not add the trailing "/" (e.g. thresholds: { 'browser_web_vital_lcp{url:https://test.k6.io}': ['p(95) < 2000'] }) which will mean that k6 will not be able to correlate the metrics with the thresholds. The end result will mean that the threshold value will be incorrect.

k6 version

NA

OS

NA

Docker version and image (if applicable)

No response

Steps to reproduce the problem

  1. Run the following:
import { browser } from 'k6/browser';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    FsFacility: {
      executor: 'per-vu-iterations',
      vus: 1,
      iterations: 1,
      options: {
        browser: {
          type: 'chromium',
        },
      },
    },
  },
  thresholds: {
    'browser_web_vital_lcp{url:https://test.k6.io}': ['p(95) < 2000'],
  },
};

export default async function () {
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto('https://test.k6.io');
    sleep(10); // Allows time to capture LCP and other web vitals
  } finally {
    await page.close();
  }
}

Which will result in:

browser_web_vital_lcp..........: avg=204.4ms min=204.4ms med=204.4ms max=204.4ms  p(90)=204.4ms  p(95)=204.4ms

     ✓ { url:https://test.k6.io }...: avg=0s      min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s

Changing the threshold to:

thresholds: {
    'browser_web_vital_lcp{url:https://test.k6.io/}': ['p(95) < 2000'],
  },

Fixes the issue.

Expected behaviour

The threshold correlates with the metric regardless of the missing trailing "/".

Actual behaviour

The threshold doesn't correlate with the metric when it is missing the trailing "/".

ankur22 commented 2 weeks ago

It's possible to hit this issue when the page redirects to another page. E.g. when navigating to your grafana stack (some-name.grafana.com), it will redirect to the login page first. In some cases it may not redirect back to the initial page. The thresholds will not correlate due to the redirects. It currently requires the user to know up front all the possible urls that the test will navigate to and adding the thresholds in for each url, e.g.:

    thresholds: {
        'browser_web_vital_lcp{url:https://some-name.grafana.net/login}': ['p(95) < 2000'],
        'browser_web_vital_lcp{url:https://some-name.grafana.net/}': ['p(95) < 2000']
      },