Set-up benchmarking against other engines

Razican commented 2 years ago

Currently, we run benchmarks to improve (or at least not regress) on the performance of Boa. This is a very good approach, but we are missing proper benchmarking against other JavaScript engines.

Particularly, it would be interestint to benchmark against

Spidermonkey
V8 (both in Node and in Deno)

Any other possible engine would also fit. The idea is to store these comparisons for each commit, the same way we do it for our own benchmarks. We can either create new benchmarks, or try to replicate our benchmarks in those engines, and store their values on each commit.

jasonwilliams commented 2 years ago

Just dumping some ideas..

Hyperfine seems like it would be good for this, and maybe a separate repo under the boa-dev umbrella which can periodically run.

Something like test262 would be too much, so would need to be just a couple of JS scripts, like 3 or 4 max for now.

MDFL64 commented 2 years ago

This repo is pretty old but it contains a bunch of benchmark scripts that should be usable.

wusyong commented 2 years ago

This repo is pretty old but it contains a bunch of benchmark scripts that should be usable.

quickJS benchmark with v8-v7. Maybe we can see some results between boa, quickJS, and v8.

wusyong commented 2 years ago

I just modify the benchmark above so it can work with boa. Here's the result from @CYBAI https://gist.github.com/wusyong/bae2a336ee22885c4e1eed02d748f118

Richards: 8.64
DeltaBlue: 7.19
RayTrace: 30.0
----
Score (version 7): 12.3

And here's V8's:

Richards: 26212
DeltaBlue: 60518
RayTrace: 65267
----
Score (version 7): 46956

I haven't tested the rest because some of them will stuck.

jasonwilliams commented 2 years ago

Hey @wusyong great work on this, I had some questions. Are there any current benchmarks the engines are being judged by today that Boa can be added to or is this the best option? What does those numbers mean and are those names [Richards] just some test cases? What does the score mean?

wusyong commented 2 years ago

I don't this is the best benchmark tbh. This benchmark has been archived by mozilla. I think QuickJS choose this just for quick result. It doesn't provide metrics to keep monitor ever since. But V8 results from @CYBAI is similar to QuickJS posted. We can get some idea about the gaps between those engines.

gkorland commented 1 year ago

@jasonwilliams @wusyong I think that if boa is around these numbers

Richards: 8.64
DeltaBlue: 7.19
RayTrace: 30.0
----
Score (version 7): 12.3

Then there is still a long way to go here compared to the numbers QuickJS published

lastmjs commented 1 year ago

I posted this in the Discord on November 11, 2022, I think it will be useful for this issue:

I just finished independently running the benchmarks myself, let me share the numbers now. It does look like there's a 25-30x hit in performance if you compare Boa to QuickJS.

Here's QuickJS and various other engines compared with each other: https://bellard.org/quickjs/bench.html

Boa V8 benchmark results:

PROGRESS Richards
RESULT Richards 23.0
PROGRESS DeltaBlue
RESULT DeltaBlue 21.6
PROGRESS Encrypt
PROGRESS Decrypt
RESULT Crypto 32.9
PROGRESS RayTrace
RESULT RayTrace 88.1
PROGRESS Earley
PROGRESS Boyer
RESULT EarleyBoyer 95.3
ERROR RegExp TypeError: not a constructor
undefined
PROGRESS RegExp
PROGRESS Splay
RESULT Splay 114
PROGRESS NavierStokes
RESULT NavierStokes 8.05
SCORE 38.5
Uncaught Error: Benchmark had 1 errors

QuickJS V8 benchmark results:

PROGRESS Richards
RESULT Richards 799
PROGRESS DeltaBlue
RESULT DeltaBlue 789
PROGRESS Encrypt
PROGRESS Decrypt
RESULT Crypto 964
PROGRESS RayTrace
RESULT RayTrace 1065
PROGRESS Earley
PROGRESS Boyer
RESULT EarleyBoyer 1480
PROGRESS RegExp
RESULT RegExp 265
PROGRESS Splay
RESULT Splay 2023
PROGRESS NavierStokes
RESULT NavierStokes 1911
SCORE 998

It's very easy to run the benchmarks.

You need this archive: https://bellard.org/quickjs/quickjs-extras-2021-03-27.tar.xz

To run quickjs V8 benchmarks: Check out their github repo and copy the bench-v8 from the archive to tests/bench-v8 and run make bench-v8

To run boa V8 benchmarks: Check out the github repo, build using cargo build --release --bin boa, then call boa on the combined.js from the quickjs archive

jasonwilliams commented 1 year ago

Thanks for this @lastmjs its quite informative. So I think we need to do a few things here:

It would be good to have some benchmarking framework that builds on this, something we could run on CI maybe.
We’re aware we’re slower than other engines, we lack inline caching, hidden classes, and a Just In Time compiler, plus various other things. We should start tracking the progress of these more closely. Maybe even set up a project to group these by.

Let me know if there’s any of these you’d like to collaborate on it any.

I understand how important performance is and we’ll hopefully chat about it when we’re next together.

lastmjs commented 1 year ago

I can setup the benchmarking CI using the benchmarking framework I shared if that's what we'd like to do. It should be relatively simple and would at least get us started with measuring.

Also, for my company's purposes we really need optimizations besides JIT because we're executing in a Wasm runtime. There should be ~30x improvement without JIT.

jasonwilliams commented 1 year ago

@lastmjs it would be good to see https://github.com/GoogleChromeLabs/jsvu be used so we can test against engines other than quickjs

here is an example https://github.com/mathiasbynens/string-prototype-replace-regexp-benchmark/blob/main/bench.sh

would be good to see something like hyperfine used too

lastmjs commented 1 year ago

I'm suggesting using the V8 benchmarks, are you suggesting combining those benchmarks with jsvu so that we run the benchmarks against various engines in CI? Or are you suggesting another benchmarking framework be used?

The simplest thing to start with would be getting the V8 benchmarks to run in CI, so we can compare against this chart: https://bellard.org/quickjs/bench.html

jasonwilliams commented 1 year ago

are you suggesting combining those benchmarks with jsvu so that we run the benchmarks against various engines in CI?

Yes I'm suggesting this. Do you have a link to the V8 benchmarks?

lastmjs commented 1 year ago

This is all done with the V8 benchmarks: https://bellard.org/quickjs/bench.html

Everything I have shared so far has been using the V8 benchmarks.

jasonwilliams commented 7 months ago

In order for us to keep on track with making sure we're focusing on performance i propose we retire the current benchmarks we have and migrate to the V8 benchmarks quickjs were using. Funnily enough quickJS have moved away from that benchmark suite now to something they have built-in but I still think it should make for a better benchmark that we have today.

The current suite is no longer useful or serving its purpose, it was created a long time ago and there have been so many optimizations since then that the numbers on the graphs are so out of magnitude you can't make use of it today. Also the things we were benchmarking or checking have either changed or we've refactored so we (the maintainers) don't look at that page too often. Although the v8 benches are no longer used by V8 themselves they will still serve a useful purpose for us at least until we reach a point where we're much closer to the other engines.

I would propose us having a nightly job which can take the bench-v8 from the archive like @lastmjs showed and run the latest boa vs other engines from esvu or eshost then print the results into a JSON file on the data repo. The website can take that data and generate some graphs with it.

We can start of with this job running as a Github action, but if its too noisy we can move it to a dedicated server.