evanwashere / mitata

benchmark tooling that loves you ❤️
MIT License
1.51k stars 23 forks source link

Please bring back group names. #31

Closed arthurfiorette closed 1 week ago

arthurfiorette commented 1 month ago

When running multiple benchs in the same run, its now very hard to distinguish the output into which run is for which group.

For example, I'm running the same bench for 1, 500, 1000, 2000, 5000, 10000 number of childs, previously i used the group name to say which group (boxplot now - btw this is amazing), and now there's no way to separate them into the output, since if i write something like the code below, it will be printed before even the first benchmark run:

for (const runs of [1, 500, 1000, 2000, 5000, 10000]) {
  const data = generate(runs);

  boxplot(() => {
    // this will be printed before any benchmark runs
    console.log('Runs:', runs);

    for (const fn of functions) {
      bench(fn.name, () => fn(data));
    }
  });
}
evanwashere commented 1 month ago

group names had to go for sake of matching capabilities of google/benchmark regex filter and ensuring json output usefulness for upcoming history comparing tool

this is how you would make it part of benchmark name:

boxplot(() => {
  for (const fn of functions) {
    bench(`${fn.name}($runs)`, function* (ctx) {
      const data = generate(ctx.get('runs'));
      yield () => fn(data);
    })
      .args('runs', [1, 500, 1000, 2000, 5000, 10000]);
  }
});

// or (matches your current boxplot output)
for (const runs of [1, 500, 1000, 2000, 5000, 10000]) {
  boxplot(() => {
    for (const fn of functions) {
      bench(`${fn.name}(${runs})`, function* (state) {
        const data = generate(runs);
        yield () => fn(data);
      });
    }
  });
}
cometkim commented 2 weeks ago

I wrote my benchmarks heavily relies on groups, and I can't get the same thing in v1. e.g. https://github.com/cometkim/unicode-segmenter/blob/53aef11/benchmark/grapheme/perf.js

It would be nice if there is at least some hooks that can be used for printing the delimiters myself.

evanwashere commented 1 week ago

I will add it back but it will only be visual (no regex filtering)

cometkim commented 1 week ago

It sounds perfect to me

evanwashere commented 1 week ago

group names are back in ^1.0.12