Shopify / yjit-bench

Set of benchmarks for the YJIT CRuby JIT compiler and other Ruby implementations.
MIT License
87 stars 22 forks source link

Increase rack benchmark size #296

Closed casperisfine closed 4 months ago

casperisfine commented 4 months ago

Currently the rack benchmark really doesn't do a whole lot. It's a single rack application with just the rack router, so it's basically a handful of method calls and hash lookups.

Even on the interpreter, it's ~40ms for 10k requests per iteration, that's really not a whole lot of code to exercise for a benchmark.

By adding a bunch more standard middlewares that most applications in the while actually do run, we can make the benchmark a bit more relevant.

Before:

interp: ruby 3.3.1 (2024-04-23 revision c56cd86388) [arm64-darwin23]
yjit: ruby 3.3.1 (2024-04-23 revision c56cd86388) +YJIT [arm64-darwin23]

-----  -----------  ----------  ---------  ----------  ------------  -----------
bench  interp (ms)  stddev (%)  yjit (ms)  stddev (%)  yjit 1st itr  interp/yjit
rack   42.0         1.9         35.0       4.9         1.13          1.20
-----  -----------  ----------  ---------  ----------  ------------  -----------

After:

interp: ruby 3.3.1 (2024-04-23 revision c56cd86388) [arm64-darwin23]
yjit: ruby 3.3.1 (2024-04-23 revision c56cd86388) +YJIT [arm64-darwin23]

-----  -----------  ----------  ---------  ----------  ------------  -----------
bench  interp (ms)  stddev (%)  yjit (ms)  stddev (%)  yjit 1st itr  interp/yjit
rack   225.2        1.1         191.8      2.1         1.15          1.17
-----  -----------  ----------  ---------  ----------  ------------  -----------
maximecb commented 4 months ago

Thanks Jean. We still get very little speedup. Is rack mostly a native gem?

casperisfine commented 4 months ago

Is rack mostly a native gem?

No, it's pure Ruby. But it doesn't do a whole lot except simple method calls and hash lookups.

I'll profile it.

casperisfine commented 4 months ago

Lol:

Capture d’écran 2024-05-07 à 16 43 24

I'll see if this Dir.pwd is strictly required. Or maybe just remove that middleware.

casperisfine commented 4 months ago

https://github.com/Shopify/yjit-bench/pull/298

casperisfine commented 4 months ago

Figured it out: https://github.com/Shopify/yjit-bench/pull/299