High cardinality on URL labels with using the Xk6 extension for prometheus remote write

[perrinj3]

I am using the latest code to build the promremotewrite extension for K6 and have encountered some behaviour we don't see with K6 core code.

We are using K6 to performance test at a high rate with a large number of time series. We also have a range of labels specified. What we see when we query Prometheus is a label called url which has a high number of values as we are using dynamic urls in our testing. We do not specify url as a label in the script. If we specify url it has no impact on this behaviour Due to the high cardinality on this url label we get severe performance issues in Prometheus. (serveral hundre thousand values) In the K6 core version we note that the name label needs to be specified as it defaults to the url value otherwise But with the k6 prom extension we no longer get a name label although we have specified it and URL now appears as a label. We tried overriding the URL label but it has no effect.

Expected behaviour url does not appear as a label in the k6 metric as this can cause high cardinality issues with dynamic URLs name label when specified in the k6 script should appear as a label in Prometheus

Observed behaviour url appears as a label in the k6 metric when queried in Prometheus. This causes performance issues due to high cardinality name label does not appear in Prometheus even though it is specified in the script

Sample script

import http from 'k6/http';
import { sleep } from 'k6';
import { Rate } from 'k6/metrics';
import { Counter } from 'k6/metrics';

export default function () {

  let res1 = http.get('http://metrics-observability-metric-engcoeliberty-dev-di1001.apps-int.di1001.cpaas.test/', {
    tags: { type: 'API Call' ,group: 'Login', name: 'Directory',scenario: 'Smoke Test1', name: 'name3'},
  });

  let responses = http.batch([
    [
      'GET',
      'http://metrics-observability-metric-engcoeliberty-dev-di1001.apps-int.di1001.cpaas.test/foo',
      null,
      { tags: { type: 'staticContent',group: 'Folder' ,name: 'Directory2', scenario: 'Smoke Test1', name: 'name1'} },
    ],
    [
      'GET',
      'http://metrics-observability-metric-engcoeliberty-dev-di1001.apps-int.di1001.cpaas.test/bar',
      null,
      { tags: { type: 'staticContent',group: 'WebServer',name: 'Dir List2', scenario: 'Batch Test1', name: 'name2' } },
    ],

  ]);

  sleep(1);
}

[na--]

I am not very familiar with this extension yet, but looking at its config code, it seems like for some reason it keeps the url and not the name tag by default :confused: I am not sure why that was, but you should be able to reverse it by starting the bundled k6 binary with the K6_KEEP_NAME_TAG=true K6_KEEP_URL_TAG=false environment variables?

FWIW, we are also working on solving this issue more generally in upcoming k6 versions, see this section in the latest k6 v0.39.0 release notes.

[perrinj3]

Thanks, the K6_KEEP_NAME_TAG=true K6_KEEP_URL_TAG=false environment variables work as suggested. Also thanks for the link on upcoming changes. We'll keep a close eye on this. It'll be good to have the prom writer in the core