Open BioPhoton opened 6 months ago
Sure, @sirenkovladd might be interested?
Also relevant in context of vitest https://github.com/vitest-dev/vitest/issues/5412
Tasks are already running sequentially. https://github.com/tinylibs/tinybench/blob/main/test/sequential.test.ts
So let me know what is the specific issue! As far as I know, vitest does not use bench.run from tinybench and it implements its own way.
@BioPhoton Do you have any reproduciton?
I have the following suite:
// eslint-ignore-next-line import/no-named-as-default-member
import fastGlob from 'fast-glob';
import { glob } from 'glob';
import { globby } from 'globby';
import { join } from 'node:path';
import yargs from 'yargs';
const cli = yargs(process.argv).options({
pattern: {
type: 'array',
string: true,
default: [
join(process.cwd(), 'node_modules/**/*.md'),
],
},
outputDir: {
type: 'string',
},
logs: {
type: 'boolean',
default: true,
},
});
// eslint-disable-next-line n/no-sync
const { pattern, logs } = cli.parseSync();
if (logs) {
// eslint-disable-next-line no-console
console.log('You can adjust the test with the following arguments:');
// eslint-disable-next-line no-console
console.log(
`pattern glob pattern of test --pattern=${pattern.toString()}`,
);
}
const fastGlobName = 'fast-glob';
const globName = 'glob';
const globbyName = 'globby';
// ==================
const suiteConfig = {
suiteName: 'glob-matching',
cases: [
[fastGlobName, callAndValidate(fastGlob.async, pattern, fastGlobName)],
[globName, callAndValidate(glob, pattern, globName)],
[globbyName, callAndValidate(globby, pattern, globbyName)],
]
};
export default suiteConfig;
// ==============================================================
const logged: Record<string, boolean> = {};
function callAndValidate<T = string | string[]>(
fn: (patterns: T) => Promise<unknown[]>,
globPatterns: T,
fnName: string,
) {
return async () => {
const result = await fn(globPatterns);
if (result.length === 0) {
throw new Error(`Result length is ${result.length}. This test is not meaningful`);
} else {
if (!logged[fnName]) {
logged[fnName] = true;
console.log(
`${fnName} found ${result.length} files for pattern ${pattern.join(
', ',
)}`,
);
}
}
};
}
Somehow it produces different test results than expected.
Ok, I'll check this out soon.
Problem: I wrote a benchmark to compare file crawling and get quite different results compared to e.g. benchmark. With benchmark I'm able to run tests in sequence so they can all after another access the same resources. In this way the results are reliable.
With tinybench I throw out of memory or other odd errors due to parallel processing.
Suggested solution:
Implement a way to run all test cases in sequence.
Links:
Upvote & Fund