search

Shopify / yjit-metrics

"Tasks for benchmarking, building and collecting stats for YJIT"
MIT License
14 stars 9 forks source link

Change speedup reporting to make ourselves look better #317

Closed maximecb closed 1 month ago

maximecb commented 1 month ago

The title of the issue is a bit tongue in cheek, but, I was looking at speed.yjit.org today. We're showing YJIT 3.3.4 at 1.67x of CRuby 3.3.4 and YJIT 3.4.0 at 1.74x.

At the top of the page, it says:

"Using the geomean of the headline benchmarks for x86 YJIT 3.4.0dev is 3.7% faster than YJIT 3.3.4"

But teeeechnically, maybe we could report this as, compared to YJIT 3.3.4, YJIT 3.4.0 provides an additional 7% speedup over CRuby 3.3.4?

It's just a different way of reporting the same data which highlights it in a more positive way?

Comments welcome @k0kubun @rwstauner @paracycle

rwstauner commented 1 month ago

I'm confused between the word "additional" and the "CRuby" at the end.

The current text is

YJIT 3.4.0dev is

  • 90.9% faster than CRuby 3.4.0dev
  • 4.2% faster than YJIT 3.3.4

Did you want to just change the wording (but keep the same values)?

As in:

YJIT 3.4.0dev is 90.9% faster than CRuby 3.4.0dev. Compared to YJIT 3.3.4, YJIT 3.4.0dev provides an additional 4.2% speedup over YJIT 3.3.4.

Or did you want to change our comparison of "yjit-dev to yjit-previous" to instead compare "yjit-dev to cruby-previous"?

That would be:

YJIT 3.4.0dev is 90.9% faster than CRuby 3.4.0dev. Compared to YJIT 3.3.4, YJIT 3.4.0dev provides a speedup of 88.0% over CRuby 3.3.4.

since the geomean for 3.4.0dev is still trailing the 3.3.4 baseline by 2%.

maximecb commented 1 month ago

I was thinking that we could change the calculation in terms of relative percentage increase compared to CRuby 3.3.4... But maybe this is actually silly. Most people will likely be updating from YJIT 3.3 to YJIT 3.4, and not from CRuby 3.3 to YJIT 3.4.