Closed glebm closed 6 years ago
FYI I get performance improvement from between 30% to 60% on my version, so I marked the README as 40%. You can see from my Travis CI records it recorded a 60.4% performance boost. The proof.
The performance difference swings pretty widely.
Looking at yours I see 70.8%. That's pretty impressive.
I wonder if we can somehow make the benchmark more consistent? Increase the number of iterations perhaps?
That is a possibility. One problem is when we start breaking raw metal speeds slight differences look a lot bigger unless we make the benchmarks run for a very long time. Also we've noticed differences based on system CPU cache which may vary between whats available in the system at that moment.
We might need a flag feature on the pinch bench test for --long-run
to sift out volatility. That would be great for Travis CI stats but then we have to wait that much longer for our builds to pass.
IDEA: We can make the --long-run
flag only on the master branch. This won't delay any PRs that way and we still get the stats.
Looking into it a bit more, perhaps volatility is not the issue. Here are some numbers from a Travis Linux Ruby 2.5.0 run:
64-bit ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
64-bit rustc 1.24.1 (d3ae9a9e0 2018-02-27)
architecture: x86_64-unknown-linux-gnu
--------------------------------------------------------------------------------
Performance change for allocate, instead of new, is 84.2%
Performance change for absolute? is 95.8%
Performance change for add_trailing_separator is 68.7%
Performance change for basename is 49.4%
Performance change for children is 39.2%
Performance change for children_compat is -19.6%
Performance change for chop_basename is 76.6%
Performance change for cleanpath_aggressive is 78.2%
Performance change for cleanpath_conservative is 74.1%
Performance change for del_trailing_separator is 87.7%
Performance change for directory? is 16.8%
Performance change for dirname is 28.9%
Performance change for entries is 36.1%
Performance change for entries_compat is -195.3%
Performance change for extname is 66.2%
Performance change for has_trailing_separator? is 86.3%
Performance change for join is 67.9%
Performance change for plus is 86.6%
Performance change for relative? is 87.2%
Performance change for relative_path_from is 74.0%
These are very different relative to each other from what I get locally. If Travis does virtualization with CPU sharing, perhaps that would explain the variability.
64-bit ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
64-bit rustc 1.24.1 (d3ae9a9e0 2018-02-27)
architecture: x86_64-unknown-linux-gnu
--------------------------------------------------------------------------------
Performance change for allocate, instead of new, is 82.3%
Performance change for absolute? is 93.4%
Performance change for add_trailing_separator is 40.6%
Performance change for basename is 12.1%
Performance change for children is 39.7%
Performance change for children_compat is -35.3%
Performance change for chop_basename is 73.1%
Performance change for cleanpath_aggressive is 72.7%
Performance change for cleanpath_conservative is 73.6%
Performance change for del_trailing_separator is 82.6%
Performance change for directory? is 10.0%
Performance change for dirname is 43.4%
Performance change for entries is 24.0%
Performance change for entries_compat is -63.2%
Performance change for extname is 64.1%
Performance change for has_trailing_separator? is 74.6%
Performance change for join is 63.8%
Performance change for plus is 78.4%
Performance change for relative? is 82.2%
Performance change for relative_path_from is 69.0%
Compare basename
and dirname
to the Travis numbers. My local runs are consistent within ~5% usually.
Yeah… the stats on Travis do seem much more volatile. I wonder which language is affected more by their environment/system — Is Ruby slower because of it, or Rust faster?
Seems like these variations on Travis are caused by preemption from other VM instances, so it's random.
Ah yes. The processor Spectre and Meltdown issues.
By the way the only reason I rewrote your version of extname
from earlier is when I was adding in official tests only the extname
method failed the test suite. I'm glad you were able to make this much more concise version of it now. :smile:
Based on top of #151