vitest-dev / vitest

Next generation testing framework powered by Vite.
https://vitest.dev
MIT License
12.68k stars 1.14k forks source link

The import language affect the benchmark result #6543

Open D-Sketon opened 5 hours ago

D-Sketon commented 5 hours ago

Describe the bug

I try the following code and run vitest bench

import { bench, describe } from "vitest";
import { after as afterToolkit } from "es-toolkit/function";

const after = afterToolkit;
const add = (a: number, b: number) => a + b;
const n = 10;
describe("after", () => {
  bench("es-toolkit/after - reassign", () => {
    after(n, add);
  });

  bench("es-toolkit/after", () => {
    afterToolkit(n, add);
  });
});

Here's what I got: {47802ED8-3BC5-4898-BA84-0CEFDCEF555A}

You can see that a direct import statement using import is significantly slower than reassigning values However, when I try tinybench like this:

import { Bench } from "tinybench";
import { after as afterToolkit } from "es-toolkit";

const bench = new Bench({ time: 100 });
const add = (a: number, b: number) => a + b;
const n = 10;
const after = afterToolkit;
bench
  .add("es-toolkit/after - reassign", () => {
    after(n, add);
  })
  .add("es-toolkit/after", () => {
    afterToolkit(n, add);
  });

await bench.warmup();
await bench.run();

console.table(bench.table());

There is no difference between them: {000A4F95-07BD-4389-B166-32207828BF30}

Reproduction

Use the code above

System Info

System:
    OS: Windows 11 10.0.22631
    CPU: (16) x64 13th Gen Intel(R) Core(TM) i5-13400F
    Memory: 16.18 GB / 31.82 GB
  Binaries:
    Node: 20.13.1 - D:\nodejs\node.EXE
    Yarn: 1.22.22 - D:\nodejs\yarn.CMD
    npm: 10.5.2 - D:\nodejs\npm.CMD
    bun: 1.1.27 - D:\nodejs\bun.CMD
  Browsers:
    Edge: Chromium (127.0.2651.74)
    Internet Explorer: 11.0.22621.3527
  npmPackages:
    vitest: ^2.0.4 => 2.0.4

Used Package Manager

yarn

Validations

dayongkr commented 5 hours ago

The same issue arises when calling with the subpath as shown below, resulting in a performance difference.

Before

import { bench, describe } from 'vitest';
import { after as afterToolkit } from 'es-toolkit'; // This is difference.
import { after as afterLodash } from 'lodash';

describe('after', () => {
  bench('es-toolkit/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterToolkit(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });

  bench('lodash/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterLodash(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });
});

Bench Result

image

Using subpath

import { bench, describe } from 'vitest';
import { after as afterToolkit } from 'es-toolkit/function'; // This is difference.
import { after as afterLodash } from 'lodash';

describe('after', () => {
  bench('es-toolkit/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterToolkit(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });

  bench('lodash/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterLodash(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });
});

Bench Result

image

Additional

You can see our barrel files in here(src/index.ts, src/function/index.ts).

Thanks