alexeyraspopov / picocolors

The tiniest and the fastest library for terminal output formatting with ANSI colors
ISC License
1.34k stars 44 forks source link

Rework loading benchmark to avoid v8 caching #36

Closed alexeyraspopov closed 3 weeks ago

alexeyraspopov commented 3 years ago

Benchmarks are quite random by themselves but there are things we must do to make them close to real numbers. As reported in #35, loading benchmark seems rigged because it uses a single process to test the init time of libraries, thus making the last one to test a clear winner.

In this PR I made it so the libraries are being tested via importing them in a separate node processes. It will take longer to test but the results seem to be more coherent. There is still some deviation but moving picocolors up and down does not make significant changes (comparing to the reported issue)

$ node benchmarks/loading.js
  chalk          4.687 ms
+ picocolors     0.225 ms
  cli-color     38.682 ms
  ansi-colors    1.056 ms
  kleur          1.641 ms
  kleur/colors   1.618 ms
  colorette      0.969 ms
  nanocolors     0.559 ms
$ node benchmarks/loading.js
+ picocolors     0.685 ms
  chalk          5.954 ms
  cli-color     40.953 ms
  ansi-colors    1.477 ms
  kleur          2.101 ms
  kleur/colors   1.444 ms
  colorette      1.077 ms
  nanocolors     0.704 ms
$ node benchmarks/loading.js
  chalk          5.189 ms
  cli-color     44.259 ms
  ansi-colors    1.162 ms
  kleur          1.619 ms
  kleur/colors   1.640 ms
  colorette      1.045 ms
  nanocolors     0.752 ms
+ picocolors     0.359 ms

cc @ai

Closes #35

alexeyten commented 3 years ago

Sometimes I've got negative loading time :rofl:

Benchmarking is hard…

alexeyten commented 3 years ago

I've tested some loading using child_process.fork. Here some results

https://gist.github.com/alexeyten/667e6163777783ed02879a0e3e66dd5d

alexeyraspopov commented 3 years ago

@alexeyten, seems like your approach looks better, thank you! I'll try a couple of more tests and will update this PR