Add benchmarks

[bnoordhuis]

Right now I'm ad hoc benchmarking, mostly tests/microbench.js and sunspider, but a) I'm not being very scientific about it, and b) they're micro-benchmarks.

I'd like to keep track of performance on a commit-by-commit basis. My thinking is that we run benchmarks on every commit to master and store/plot the results somewhere. Suggestions how/where very welcome.

In no particular order:

• set up sunspider, kraken and octane; maybe steal vendor V8's csuite tool: https://v8.dev/docs/benchmarks

• integrate the web tooling benchmark; it's a collection of proper macro benchmarks

• decide on metrics to track. Off the cuff: CPU time, memory, opcodes executed? GHA is probably too volatile to track wall clock time

[saghul]

Sounds cool! Regarding storing things, maybe we can store them on a separate repo, run a chron GHA and commit the result.

I'll admit I have no experience with any of those tools, but happy to take a look!

[bnoordhuis]

We did something like that for deno although I don't remember the exact details.

@littledivy if I can impose on your time, do you know offhand how that is (or was) set up? The denobot still commits to denoland/benchmark_data, it looks like.

[littledivy]

The old github actions setup in Deno is still working but yeah, GHA is quite noisy.

We switched to benchmarking on bare metal https://deploy.equinix.com/developers/docs/metal/deploy/spot-market/. Costs almost nothing. We run it after every release (4-ish in a month) and runs for 3-4hr per run.

Basically, allocates a machine via GHA cron -> runs the bench on the machine -> machine commits to a benchmark_data-like repo connected to GH pages UI.

[bnoordhuis]

Thanks! GHA noisiness is probably less of an issue for us because we're not doing I/O, only CPU/memory-bound tasks.

Okay, so I think the actions that need to be taken are:

• create bot user
• create token/secret
• create benchmark repo
• create GHA cron job that runs benchmarks and commits the results

I'd simply clone quickjs-ng/quickjs at the start of the benchmark job. Simple, and then the bot doesn't need write access to this repo, only the benchmark repo. Volunteers?

[saghul]

I think we don't even need a new user, a PAT should provide enough access control I think.

[bnoordhuis]

I can see I'm out of my depth here. :-)

(If you're wondering if I'm surreptitiously trying to get you to step forward - I do believe I am.)

[saghul]

I'll get the ball rolling! ✋

[littledivy]

Added octane, kraken and sunspider runners to https://github.com/littledivy/quickjs_ng_benchmarks (probably needs JSON export)

WDYT? Happy to transfer repo ownership

[saghul]

Nice! I'll take a look shortly!

[bnoordhuis]

@littledivy how about I create a quickjs-ng/quickjs_ng_benchmarks repo and you open a pull request so we can go over the code together? It mostly looks good but there are a few things that maybe need addressing.

[saghul]

How about quickjs-ng/benchmarks ? Do we need the long name? :-)

[littledivy]

Sounds good

[bnoordhuis]

https://github.com/quickjs-ng/benchmarks

[bnoordhuis]

I have web-tooling-benchmark more or less set up (thanks @bmeurer, it's awesome.)

The results are neither pretty nor unexpected but it's already providing valuable insights. I wouldn't have guessed we'd be spending so much time in lre_exec_backtrack() or push_state(). Lots of room for improvement there.

The benchmarks themselves are rolled up into a single 32 MB file. Anyone object to me checking that in?

edit: https://github.com/quickjs-ng/web-tooling-benchmark - checked in build artifacts, reduced sample size from 20 to 1, and disabled the coffeescript benchmark

$ ../quickjs/build/qjs --stack-size 2000000 dist/cli.js                                                              
Running Web Tooling Benchmark v0.5.3…                                          
-------------------------------------                                          
         acorn:  0.41 runs/s                                                                                                                                   
         babel:  1.12 runs/s                                                                                                                                   
  babel-minify:  1.44 runs/s                                                   
       babylon:  0.79 runs/s                                                                                                                                   
         buble:  1.46 runs/s                                                                                                                                   
          chai:  2.16 runs/s                                                   
        espree:  0.26 runs/s                                                                                                                                   
       esprima:  0.54 runs/s                                                                                                                                   
        jshint:  1.26 runs/s                                                                                                                                   
         lebab:  1.14 runs/s                                                                                                                                   
       postcss:  0.69 runs/s                                                   
       prepack:  1.33 runs/s                                                                                                                                   
      prettier:  0.48 runs/s                                                                                                                                   
    source-map:  0.69 runs/s                                                   
        terser:  3.18 runs/s                                                                                                                                   
    typescript:  1.75 runs/s                                                                                                                                   
     uglify-js:  0.88 runs/s                                                                                                                                   
-------------------------------------                                                                                                                          
Geometric mean:  0.96 runs/s                      

[saghul]

Great stuff!

[littledivy]

@bnoordhuis Can you create a default branch in https://github.com/quickjs-ng/benchmarks? Need a base branch to send a PR.

[saghul]

Done!