search

bytecodealliance / sightglass

A benchmark suite and tool to compare different implementations of the same primitives.
Apache License 2.0
69 stars 33 forks source link

Use `all.suite` to run all benchmarks #254

Closed abrown closed 1 year ago

abrown commented 1 year ago

If benchmarks are no longer all named benchmark.wasm, then we need some other way than the benchmarks/*/benchmark.wasm glob to list out all benchmarks. This change creates an all.suite file which contains all known benchmark paths. In order to keep this up-to-date as new benchmarks are added, it also adds a new CI action to run a script checking that all.suite contains all benchmark paths. Note that some benchmarks (like image-classification, which requires extra setup) are commented out--the check-incomplete-suite.sh script is smart enough to ignore these. Finally, this change updates documentation to use all.suite instead of benchmarks/*/benchmark.wasm in several places.

abrown commented 1 year ago

Do not merge yet; this is a follow-on to #252. As a draft it demonstrates that there is still a way to run all of the benchmarks after #250 without using the benchmarks/*/benchmark.wasm glob.

abrown commented 1 year ago

2 hours and 30 minutes is a bit excessive to wait to run all the benchmarks. Some thoughts: dial down the libsodium iterations or only run a subset of benchmarks.

abrown commented 1 year ago

2 hours and 30 minutes is a bit excessive to wait to run all the benchmarks. Some thoughts: dial down the libsodium iterations or only run a subset of benchmarks.

The latest commit to this branch, b4bec91, dials the number of libsodium iterations down to 1. The idea is to use Sightglass to configure the number of iterations instead. And doing this vastly reduces the time it takes to run all of the benchmarks.