search

ewasm / benchmarking

Instructions for benchmarking Ewasm precompiles. (And results.)
Apache License 2.0
18 stars 9 forks source link

Document the version/commithash/date or each VM used #41

Closed axic closed 4 years ago

axic commented 4 years ago

Can start collecting that here in the issue, but eventually should be documented in the readme or similar.

The hash is the upstream hash the branch is based on.

axic commented 4 years ago

Suggestion: rebase these branches on top of the closest release and name them as <version>-benchmarking, e.g. wasm3 would have a branch called v0.4.2-benchmarking based on the v0.4.2 release.

hugo-dc commented 4 years ago

Here is the list of versions of VMs used:

axic commented 4 years ago

@jwasinger @hugo-dc rebased the wasmi commit on wasmi 0.4.3 and moved the patch from examples/invoke.rs to src/bin/invoke.rs. This means it can be built with cargo build --release without requiring the tests and wabt to build...

It is available at https://github.com/ewasm-benchmarking/wasmi/tree/v0.4.3-benchmarking

jwasinger commented 4 years ago

btw I tried that wasmi branch and I get an error:

vms_to_run: {'wasmi': <project.VMDescriptor.VMDescriptor object at 0x7fecf4d629d0>}
06/08/2020 04:23:45 AM <wasm_bencher>: launch guido-fuzzer-find-2 test
06/08/2020 04:23:45 AM <wasm_bencher>: /engines/wasmi/invoke /wasmfilesminified/guido-fuzzer-find-2.wasm main
['/engines/wasmi/invoke', '/wasmfilesminified/guido-fuzzer-find-2.wasm', 'main']
stdout_lines:  thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
compile_lines:  thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
compile_match:  None
06/08/2020 04:23:45 AM <wasm_bencher>: wasmi (/engines/wasmi/invoke /wasmfilesminified/guido-fuzzer-find-2.wasm main) got exception:
Error parsing engine output. exception: 'NoneType' object is not subscriptable
engine output:
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
06/08/2020 04:23:45 AM <wasm_bencher>: skipping engine wasmi for bench input guido-fuzzer-find-2
06/08/2020 04:23:45 AM <wasm_bencher>: launch bls12-381-aggreg-64-pubkeys-verify-sig test
06/08/2020 04:23:45 AM <wasm_bencher>: /engines/wasmi/invoke /wasmfilesminified/bls12-381-aggreg-64-pubkeys-verify-sig.wasm main
axic commented 4 years ago

Also pushed wasm3 v0.4.7-benchmarking: https://github.com/ewasm-benchmarking/wasm3/tree/v0.4.7-benchmarking

axic commented 4 years ago

btw I tried that wasmi branch and I get an error:

Weird. We can try based those commits on v0.4.4: https://github.com/ewasm-benchmarking/wasmi/tree/v0.4.4-benchmarking

axic commented 4 years ago

We should also use git tags and not git branches on ewasm-benchmarking/<repo>.

The reason: those can not be changed (well, not as easily as branches), so the benchmarking is way more reproducible.

axic commented 4 years ago

New suggestion:

  1. Keep a benchmarking branch on each of the forks with the code.
  2. Tag each release, i.e.v1.2.3-benchmarking.

The reason: the tags can't be modified easily, but the branches are easily noticeable if we want to work on them later.

axic commented 4 years ago

Updates:

  1. Most of the easy pickings and/or useful ones were done: ssvm, wasm3, evmone, cita-vm, and parity

  2. Two are still pending: wasmi and wagon

  3. The following I don't think is worth bothering with given it is a lot of commits apart: wavm

  4. We are not touching wabt, but rather renaming it, and adding vanilla-wabt (some recent version)

  5. Still need to investigate: wamr and wasmtime

axic commented 4 years ago

I tried figuring out the a closeby release for wasmtime but I think it is just too far apart. Would be nice to use a newer wasmtime, but for now I suggest we leave it alone.

Closing this and opening a tracking issue for wasmtime.