Publicly viewable benchmarks against native?

[kirawi]

Feature

I would like it if Wasmtime would publish benchmarks of native Rust code versus it compiled down to Wasm.

Benefit

It would help decide how performant it is compared to a direct scripting language, and if its portability outweighs its worse performance. There isn't a source truth for such comparisons AFAIK except https://00f.net/2021/02/22/webassembly-runtimes-benchmarks/

Implementation

Outside of my knowledge.

Alternatives

Don't implement/publish such benchmarks.

[cfallin]

Some folks have been working on benchmarking infrastructure using Sightglass with possibly some CI integration and/or historical views -- @jlb6740 what's the latest status on that?

[kirawi]

It doesn't look like it benchmarks the native equivalent.

[cfallin]

In the past, at least in presentations I've seen, Sightglass results have included performance ratios against the same code running natively; and I think that's the plan going forward, but @jlb6740 or @fitzgen could comment further?

[fitzgen]

The benchmark suite is explicitly interally focused (is this PR a regression/improvement? what is our perf trend over time?) and we de-emphasized external presentation, "benchmarketing", and direct comparisons to other wasm engines or native, per the RFC.

@kirawi You could probably get the benchmarks running on native without a ton of work if you're motivated to do that, but it isn't really the focus of our benchmarking efforts.

[kirawi]

I think it would still be valuable to have benchmarks against native code to help in deciding whether a project like Wasmtime fits design requirements. Wasm will obviously be overall slower, but how slow is a big deciding factor.