False negative detecting WebAssembly

[hexpunk]

Browser: Chrome Version: 57.0.2987.110 (64-bit)

---

Computer: MacBook Pro (Retina, 15-inch, Mid 2015) Processor: 2.5 GHz Intel Core i7 Memory: 16 GB 1600 MHz DDR3 Graphics: Intel Iris Pro 1536 MB

---

OS: OSX/macOS Version: Sierra 10.12.3 (16D32)

---

Trace (DevTools console):

requesting benchmark box2d-throughput 3
benchmark-worker.js:30 using asmjs
driver.js:684 {"benchmark":"box2d-throughput","runtime":2662,"average":6.141,"variance":6.009,"lowest":4,"highest":116}
driver.js:697 requesting benchmark box2d-throughput 3
benchmark-worker.js:30 using asmjs
driver.js:684 {"benchmark":"box2d-throughput","runtime":2590,"average":6.164,"variance":5.898,"lowest":4,"highest":115}
driver.js:697 requesting benchmark box2d-throughput 3
benchmark-worker.js:30 using asmjs
driver.js:684 {"benchmark":"box2d-throughput","runtime":2834,"average":6.753,"variance":6.489,"lowest":4,"highest":126}
driver.js:697 requesting benchmark box2d-throughput 3
benchmark-worker.js:30 using asmjs
driver.js:684 {"benchmark":"box2d-throughput","runtime":2782,"average":6.645,"variance":6.327,"lowest":4,"highest":123}
driver.js:636 final: [{"benchmark":"box2d-throughput","runtime":2662,"average":6.141,"variance":6.009,"lowest":4,"highest":116},{"benchmark":"box2d-throughput","runtime":2590,"average":6.164,"variance":5.898,"lowest":4,"highest":115},{"benchmark":"box2d-throughput","runtime":2834,"average":6.753,"variance":6.489,"lowest":4,"highest":126},{"benchmark":"box2d-throughput","runtime":2782,"average":6.645,"variance":6.327,"lowest":4,"highest":123}]
driver.js:668 final: {"runtime":2717,"average":6.42575,"variance":6.18075,"lowest":4,"highest":120} on 4
driver.js:697 requesting benchmark box2d-wasm-throughput 3
benchmark-worker.js:30 using native-wasm
box2d.js:337 no native wasm support detected
printErr @ box2d.js:337
doNativeWasm @ box2d.js:122
integrateWasmJS @ box2d.js:202
(anonymous) @ box2d.js:386
onmessage @ benchmark-worker.js:64
benchmark-worker.js:64 Uncaught Error: Uncaught no wasm method succeeded
    at onmessage (benchmark-worker.js:64)

---

As you can see in the above trace, MASSIVE isn't detecting WebAssembly support despite the fact that when I go to http://webassembly.org/demo/, it runs as expected without any note about missing WebAssembly support.

[kripken]

Yeah, massive hasn't been updated with recent wasm changes. I'm not sure it's worth doing right now, as we also need to change massive in some ways (for async startup, we need to measure startup differently) - probably we should rethink how we benchmark as a whole.

[drola]

What would be the simplest way to get box2d benchmark working in wasm mode? Is it just a matter of recompiling with new version of emscripten?

[kripken]

By chance I just finished updating box2d.js to wasm, see the wasm branch in that repo, it now has both asm.js and wasm builds. Just using them here should work - but as mentioned above, just for throughput, startup measurements may be meaningless.

[drola]

Thanks. I tried that but I looks like I am missing some source code or trying to do the wrong stuff. If I just use files from the new box2d.js build, I don't get any output into Module.printBuffer. I suspect there should be some code that runs the actual benchmark and prints a string like "frame averages: 31.675 +- 7.808, range: 22.000 to 63.000" but I am unable to find that code. Do you have any hints perhaps?

[kripken]

Hmm, yes, reading driver.js, it says https://github.com/kripken/Massive/blob/master/driver.js#L113 "let the emscripten benchmark suite generate it for you". So running something like python tests/runner.py benchmark.test_zzz_box2d in emscripten should emit it.

If we create some JS to run the benchmark, and use box2d.js, that would also measure the overhead of calling from JS into compiled code, which might or might not make sense to do, depends what you want.

[kripken]

There are builds from the emscripten benchmark here, btw, including box2d https://github.com/kripken/embenchen/tree/master/asm_v_wasm

[techtonik]

no native wasm support detected  box2d.js:337:4

Doesn't detect it on Firefox 55.

[blixt]

This test is one of the top Google results for "performance webassembly" as of March, 2018. However, it does not work in any of the up-to-date browsers that I tested (Chrome 65, Firefox 59, Safari iOS 11.2.6). It also doesn't at any point signal to the user that the test has given up (only "running...")

Maybe an update to add an alert to the top of the page to not use this test is in order?

[kripken]

@blixt we should update the docs to mention that, yeah. But which page do you mean? One of the massive pages (which?) or the asm_v_wasm one from the last few comments?

[blixt]

@kripken Sorry, the only link that I can find via Google appears to be http://kripken.github.io/Massive/beta/

Here's the queries I tried on Google:

• webassembly performance (no. 4)
• webassembly benchmark (no. 2)
• webassembly speed (no. 3)
• wasm performance (no. 3)
• wasm benchmark (no. 2)
• wasm speed (no. 2)

[kripken]

Ah thanks @blixt , yeah, we have that "beta" page up. I think I'll just remove it, until we figure out a proper benchmarking method.