WASM Binary Benchmarks

[biggusbeetus]

Hey @sbc100, I was going to re-open the past issue #17937, but I felt that it would be getting out of topic, hence a new issue.

The reason that I am asking for a compiled libstdc++ is because I am trying to compile the programs from the Computer Language Benchmark Game to wasm binaries (plus their JS /HTML glue code). However, these programs rely on external APIs/libraries, such as OpenMP, and libstdc++, and SIMD instructions, which are still not fully supported e.g. the AVX 256-bit instruction set. I could modify the benchmark implementations, but that could introduce possible bias. I know that some researchers used the SPEC CPU 2006 and 2017 benchmarks, but they had to basically extend Browsix to run wasm binaries to objectively measure the performance of I/O intensive, C compiled wasm applications. However, this project is already out of date, and honestly seems like a huge undertaking to even update the project at this point to just run some benchmarks. Besides, wasm has more use cases than I/O operations.

This begs the question, are you by any chance aware of any benchmarks made specifically for wasm?

Maybe it would be interesting to create a set of benchmarks that evolve alongside wasm, that can be used to objectively measure the performance/resource consumption of wasm binaries. It might seem obvious at first glance that statically typed, compiled languages like C are more efficient than interpreted languages like Typescript when targeted to wasm, but it would be nice to have empirical evidence. It might not even be the case given the maturity of wasm and the fact that a lot of the architecture specific SIMD instructions or parallel programming APIs like OpenMP have to be simulated/ported to wasm. Either way, any help or input would be greatly appreciated.

[sbc100]

As you know benchmarking can be a very complex and controversial area. I would be wary of making any assertions about the performance of C/C++ code built to WebAssembly compared to that of JavaScript in the general case. If you try to compare these two I don't think you will find one being faster than the other and that it really depends on the domain of you program/application.

Having said that if you are interesting in comparing the relative performance C/C++ code compiled to native vs compiled with wasm, or of comparing different compilers and runtimes targetting wasm we do have some benchmarks that we store in the emscripten repo here: https://github.com/emscripten-core/emscripten/blob/main/test/test_benchmark.py

[kripken]

Just to add to that, those benchmarks contain a bunch of real-world code (physics engines, compression libraries, etc.), so I consider it to be representative of real-world performance. For example, wasmbox benchmarking investigates the overhead of wasm sandboxing. You can also compare wasm in a VM to native builds (last time I ran that, wasm VMs were 40% slower than native).

It might seem obvious at first glance that statically typed, compiled languages like C are more efficient than interpreted languages like Typescript when targeted to wasm, but it would be nice to have empirical evidence. It might not even be the case given the maturity of wasm and the fact that a lot of the architecture specific SIMD instructions or parallel programming APIs like OpenMP have to be simulated/ported to wasm

Well, I don't think we need more evidence that C is faster than TypeScript in general. First, there is an overwhelming amount of evidence for that, but perhaps more importantly we understand why it is faster. And when an exception happens and JS/TS are faster than C then we also know why that is (e.g., regex benchmarks - because JS VMs have the fastest regex engines in the entire industry, that even JIT compile; or inlining - JS does runtime inlining that in rare cases can make all the difference).

As for wasm, as I said around 40% overhead compared to native is what benchmarks tend to show, so C compiled to wasm will generally still be faster than JS/TS on real-world code. But, again, exceptions exist, and one of them is doing a lot of DOM calls - for that reason JS/TS can end up faster than wasm for general-purpose web code. But wasm shines on pure computations like a physics engine.

[biggusbeetus]

As for wasm, as I said around 40% overhead compared to native is what benchmarks tend to show, so C compiled to wasm will generally still be faster than JS/TS on real-world code.

Yeh, 45% faster than wasm in Firefox and 55% faster than WASM in Chrome; at least back in 2019. I am pretty sure Mozilla and Google have updated their browsers wasm runtimes.

What I am ultimately interested on is on the difference between languages when compiling to wasm binaries. So the difference between C compiled to wasm and Typescript compiled to wasm for example.