Esbuild is too fast for proper analysis

[hyf0]

Each benchmark for esbuild took 700ms and the part of esbuild only took 40ms, the rest was overhead of node.

By script

benchmark results

Running "lib_esnext_cjs_lodash-es" suite...

Progress: 100%

  build:speedy:
    0.3 ops/s, ±4.74%   | slowest, 80% slower

  build:webpack:
    0.5 ops/s, ±3.86%   | 66.67% slower

  build:esbuild:
    1.5 ops/s, ±0.80%   | fastest

Finished 3 cases!
  Fastest: build:esbuild
  Slowest: build:speedy

Running "lib_esnext_cjs_minimize_lodash-es" suite...
Progress: 100%

  build:speedy:
    0.31 ops/s, ±4.18%   | 77.7% slower

  build:webpack:
    0.26 ops/s, ±2.70%   | slowest, 81.29% slower

  build:esbuild:
    1.39 ops/s, ±3.72%   | fastest

Finished 3 cases!
  Fastest: build:esbuild
  Slowest: build:webpack

Manual

pnpm build:esbuild

> lib_esnext_esm_lodash-es@1.0.0 build:esbuild /Users/bytedance/Documents/gitcodes/benchmarks/projects/lib_esnext_cjs_lodash-es
> pnpm esbuild /Users/bytedance/Documents/gitcodes/benchmarks/fixtures/lodash-es/lodash.js --sourcemap  --bundle --outfile=/Users/bytedance/Documents/gitcodes/benchmarks/projects/lib_esnext_cjs_lodash-es/dist/esbuild.js

  dist/esbuild.js      282.3kb
  dist/esbuild.js.map  780.7kb

⚡ Done in 39ms

pnpm build:webpack

> lib_esnext_esm_lodash-es@1.0.0 build:webpack /Users/bytedance/Documents/gitcodes/benchmarks/projects/lib_esnext_cjs_lodash-es
> NODE_ENV=production webpack

asset webpack.js 623 KiB [compared for emit] [big] (name: main) 1 related asset
orphan modules 596 KiB [orphan] 639 modules
runtime modules 670 bytes 3 modules
../../fixtures/lodash-es/lodash.js + 618 modules 596 KiB [built] [code generated]

webpack 5.64.0 compiled with 3 warnings in 1287 ms

pnpm build:speedy 

> lib_esnext_esm_lodash-es@1.0.0 build:speedy /Users/bytedance/Documents/gitcodes/benchmarks/projects/lib_esnext_cjs_lodash-es
> NODE_ENV=production speedy build -c speedy.config.ts

Build completed in 2276ms

[zoolsher]

I profiled the project. Speedy is effected by the format-plugin which used babel to transform esm->cjs.

[hyf0]

@zoolsher got it.

Let me set up some esm cases.

[zoolsher]

I also created an issue in speedy. The format transformer should only use babel when setting to SystemJS target.

BTW, @underfin is writing the SystemJS plugin for swc. ;)

[hyf0]

Targeting esm

Running "lib_esnext_esm_lodash-es" suite...
Progress: 100%

  build:speedy:
    0.4 ops/s, ±1.61%   | slowest, 71.43% slower

  build:webpack:
    0.5 ops/s, ±2.21%   | 64.29% slower

  build:esbuild:
    1.4 ops/s, ±0.97%   | fastest

Finished 3 cases!
  Fastest: build:esbuild
  Slowest: build:speedy

[zoolsher]

We meet with another babel transformer T_T . This one is for es6 to es5 transform.

Set target: "es6" in config will skip this.

[hardfist]

set mode='development' for benchmark @iheyunfei , what we care is dev build speedy currently

[hyf0]

set mode='development' for benchmark @iheyunfei , what we care is dev build speedy currently

Ok. I'm creating more cases.

[hyf0]

more cases

➜  benchmarks git:(main) ✗ pnpm bench

> speedystack@1.0.0 bench /Users/bytedance/Documents/gitcodes/benchmarks
> ts-node ./benchmark-runner/go.ts

included [
  'lib_dev_to-cjs_to-es5_sourcemap_lodash-es',
  'lib_dev_to-cjs_to-es6_sourcemap_lodash-es',
  'lib_dev_to-esm_to-es5_sourcemap_lodash-es',
  'lib_dev_to-esm_to-es6_minimized_sourcemap_lodash-es',
  'lib_dev_to-esm_to-es6_sourcemap_lodash-es',
  'lib_to-cjs_to-es5_sourcemap_lodash-es',
  'lib_to-cjs_to-es6_sourcemap_lodash-es',
  'lib_to-esm_to-es6_sourcemap_lodash-es',
  'lib_to-esm_to-es6_minimized_sourcemap_lodash-es',
  'lib_to-esm_to-es6_sourcemap_lodash-es'
]

start benchmark of [lib_dev_to-cjs_to-es5_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.674 s ±  0.436 s    [User: 3.198 s, System: 0.352 s]
  Range (min … max):    2.365 s …  2.982 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      3.424 s ±  0.077 s    [User: 4.881 s, System: 0.430 s]
  Range (min … max):    3.369 s …  3.478 s    2 runs

Summary
  'pnpm 'bench:webpack'' ran
    1.28 ± 0.21 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_dev_to-cjs_to-es6_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.162 s ±  0.060 s    [User: 2.797 s, System: 0.297 s]
  Range (min … max):    2.120 s …  2.205 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      2.492 s ±  0.040 s    [User: 3.310 s, System: 0.369 s]
  Range (min … max):    2.464 s …  2.521 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     731.7 ms ±  11.9 ms    [User: 788.2 ms, System: 176.4 ms]
  Range (min … max):   723.3 ms … 740.1 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    2.96 ± 0.10 times faster than 'pnpm 'bench:webpack''
    3.41 ± 0.08 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_dev_to-esm_to-es5_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.198 s ±  0.090 s    [User: 2.844 s, System: 0.301 s]
  Range (min … max):    2.135 s …  2.262 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      2.998 s ±  0.121 s    [User: 3.763 s, System: 0.433 s]
  Range (min … max):    2.913 s …  3.084 s    2 runs

Summary
  'pnpm 'bench:webpack'' ran
    1.36 ± 0.08 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_dev_to-esm_to-es6_minimized_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      5.733 s ±  0.043 s    [User: 8.176 s, System: 0.545 s]
  Range (min … max):    5.703 s …  5.763 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      1.542 s ±  0.026 s    [User: 1.838 s, System: 0.314 s]
  Range (min … max):    1.523 s …  1.560 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     736.8 ms ±   9.8 ms    [User: 794.7 ms, System: 178.9 ms]
  Range (min … max):   729.9 ms … 743.7 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    2.09 ± 0.04 times faster than 'pnpm 'bench:speedy''
    7.78 ± 0.12 times faster than 'pnpm 'bench:webpack''

start benchmark of [lib_dev_to-esm_to-es6_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.151 s ±  0.011 s    [User: 2.769 s, System: 0.295 s]
  Range (min … max):    2.143 s …  2.159 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      1.485 s ±  0.009 s    [User: 1.769 s, System: 0.293 s]
  Range (min … max):    1.479 s …  1.492 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     754.5 ms ±  28.6 ms    [User: 787.1 ms, System: 176.9 ms]
  Range (min … max):   734.3 ms … 774.7 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    1.97 ± 0.08 times faster than 'pnpm 'bench:speedy''
    2.85 ± 0.11 times faster than 'pnpm 'bench:webpack''

start benchmark of [lib_to-cjs_to-es5_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.501 s ±  0.079 s    [User: 3.240 s, System: 0.314 s]
  Range (min … max):    2.445 s …  2.557 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      3.354 s ±  0.014 s    [User: 4.661 s, System: 0.427 s]
  Range (min … max):    3.344 s …  3.364 s    2 runs

Summary
  'pnpm 'bench:webpack'' ran
    1.34 ± 0.04 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_to-cjs_to-es6_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.444 s ±  0.006 s    [User: 3.184 s, System: 0.294 s]
  Range (min … max):    2.440 s …  2.449 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      2.509 s ±  0.066 s    [User: 3.358 s, System: 0.363 s]
  Range (min … max):    2.463 s …  2.556 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     768.1 ms ±   6.7 ms    [User: 816.3 ms, System: 181.1 ms]
  Range (min … max):   763.3 ms … 772.8 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    3.18 ± 0.03 times faster than 'pnpm 'bench:webpack''
    3.27 ± 0.09 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_to-esm_to-es6_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.447 s ±  0.010 s    [User: 3.204 s, System: 0.282 s]
  Range (min … max):    2.440 s …  2.454 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      2.624 s ±  0.052 s    [User: 3.486 s, System: 0.340 s]
  Range (min … max):    2.588 s …  2.661 s    2 runs

Summary
  'pnpm 'bench:webpack'' ran
    1.07 ± 0.02 times faster than 'pnpm 'bench:speedy''

start benchmark of [lib_to-esm_to-es6_minimized_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      5.114 s ±  0.154 s    [User: 7.455 s, System: 0.453 s]
  Range (min … max):    5.005 s …  5.223 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      1.693 s ±  0.009 s    [User: 2.059 s, System: 0.319 s]
  Range (min … max):    1.687 s …  1.699 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     895.7 ms ±  49.8 ms    [User: 919.3 ms, System: 193.2 ms]
  Range (min … max):   860.4 ms … 930.9 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    1.89 ± 0.11 times faster than 'pnpm 'bench:speedy''
    5.71 ± 0.36 times faster than 'pnpm 'bench:webpack''

start benchmark of [lib_to-esm_to-es6_sourcemap_lodash-es]

Benchmark 1: pnpm 'bench:webpack'
  Time (mean ± σ):      2.915 s ±  0.007 s    [User: 3.827 s, System: 0.340 s]
  Range (min … max):    2.910 s …  2.919 s    2 runs

Benchmark 2: pnpm 'bench:speedy'
  Time (mean ± σ):      1.672 s ±  0.028 s    [User: 2.034 s, System: 0.321 s]
  Range (min … max):    1.652 s …  1.691 s    2 runs

Benchmark 3: pnpm 'bench:esbuild'
  Time (mean ± σ):     807.5 ms ±   9.6 ms    [User: 874.4 ms, System: 166.0 ms]
  Range (min … max):   800.7 ms … 814.3 ms    2 runs

Summary
  'pnpm 'bench:esbuild'' ran
    2.07 ± 0.04 times faster than 'pnpm 'bench:speedy''
    3.61 ± 0.04 times faster than 'pnpm 'bench:webpack''

[zoolsher]

Nice going

[hardfist]

https://github.com/benchmark-action/github-action-benchmark an benchmark visualize would be more helpful