Box2d result inconsistency on Chromium and Edge

[rahulchaphalkar]

When I run this workload on Chromium and Edge, Box2d has a higher score on Edge (27,851 edge, 19,658 chromium). However, when I compile the same benchmark from its c++ source as pointed in https://github.com/kripken/Massive/issues/22 (with -O3 optimization), the result shows that Chromium is around 35% faster than Edge for box2d. The results I get are in ms/frame, haven't normalized them (4.042 edge, 2.698 chromium). I varied the no. of frames too, did not affect the result. Why is there an inconsistency between the two benchmarks? Looks like there's a difference between the 2 benchmarks, how can I find out what that is? (can't figure out from minified version).

[rahulchaphalkar]

Ah, the difference was as even though I had compiled with O3, I had preserved the function names, so it wasn't minified. That seems to be slowing it down.

[kripken]

That could definitely affect startup time. Shouldn't affect throughput though.

[rahulchaphalkar]

It turns out that number of frames used in box2d affects this. In the box2d version used in massive, its default arg is 3 (64 warmup frames, 333+64 actual timed ones). Chromium behaves bad as this number is insuficient for it to warm up i think. Increasing the number to, say around 4000 frames, makes chromium numbers better than FF and Edge, while FF and Edge show marginal improvement. Since box2d is specifically a throughput benchmark, i think it should neglect the warm-up (cold start) effect, and probably should increase the number of frames (i.e. iterations) to sufficiently high number.

[kripken]

Hmm, Box2D runs for several seconds, I think that should be more than enough time for chrome's JIT to warm up. Perhaps it is hitting a bad deoptimization cycle.