google / tachometer

Statistically rigorous benchmark runner for the web
BSD 3-Clause "New" or "Revised" License
671 stars 22 forks source link

Standalone stats API #232

Closed just-boris closed 2 years ago

just-boris commented 2 years ago

This module contains great statistic logic with auto-sampling. However, the public API is strictly requires to use command-line and selenium-webdriver.

In our use-case we already have an existing puppeteer setup, which we would like to reuse for the performance testing too.

Currently we have code like this:

import { autoSampleConditionsResolved, computeDifferences, summaryStats } from 'tachometer/lib/stats';

const a = await getResults();
const b = await getResults();

const statsDiff = computeDifferences([
  { stats: summaryStats(a) },
  { stats: summaryStats(b) },
]);

return autoSampleConditionsResolved(statsDiff, { absolute: [50], relative: [0.05] });

This works, but does not seem efficient to pull all other harness and dependencies into node_modules. What do you think about publishing the statistic part as a separate module?

aomarks commented 2 years ago

I don't think we want to vend a pure statistics library, that's out of scope for us.

just-boris commented 2 years ago

Would it be okay, if I extract this code and publish it under my name (preserving the original references, of course)?