search

Shopify / yjit-bench

Set of benchmarks for the YJIT CRuby JIT compiler and other Ruby implementations.
MIT License
87 stars 22 forks source link

Let erubi-rails loop more in each iteration #256

Closed k0kubun closed 1 year ago

k0kubun commented 1 year ago

The stddev of erubi-rails is so much higher than all other headline benchmarks. As a headline benchmark, I want it to produce reliable results. The duration of each iteration in erubi-rails is so slow that it could be spending non-trivial time in the harness's loop implementation too.

To address that, this PR increases the loop count of erubi-rails so that the duration of each iteration becomes closer to other headline benchmarks. I used railsbench and lobsters to pick a target duration.

Before

interp: ruby 3.3.0dev (2023-10-20T00:12:09Z master 985370406d) [x86_64-linux]
yjit: ruby 3.3.0dev (2023-10-20T00:12:09Z master 985370406d) +YJIT [x86_64-linux]

-----------  -----------  ----------  ---------  ----------  ------------  -----------
bench        interp (ms)  stddev (%)  yjit (ms)  stddev (%)  yjit 1st itr  interp/yjit
erubi-rails  29.3         12.2        16.5       22.7        0.37          1.78
lobsters     1470.0       1.7         963.5      2.9         0.72          1.53
railsbench   3368.8       0.4         2093.2     0.8         1.36          1.61
-----------  -----------  ----------  ---------  ----------  ------------  -----------

After

interp: ruby 3.3.0dev (2023-10-20T00:12:09Z master 985370406d) [x86_64-linux]
yjit: ruby 3.3.0dev (2023-10-20T00:12:09Z master 985370406d) +YJIT [x86_64-linux]

-----------  -----------  ----------  ---------  ----------  ------------  -----------
bench        interp (ms)  stddev (%)  yjit (ms)  stddev (%)  yjit 1st itr  interp/yjit
erubi-rails  2907.5       0.4         1679.6     1.5         1.66          1.73
lobsters     1468.8       1.8         960.2      2.8         0.71          1.53
railsbench   3360.7       0.4         2080.8     0.8         1.36          1.62
-----------  -----------  ----------  ---------  ----------  ------------  -----------

The stddev of erubi-rails is much smaller, and the duration of each iteration is shorter too, making the output less verbose and the experience closer to other headline benchmarks.