search

k0kubun / hamlit

High Performance Haml Implementation
https://rubygems.org/gems/hamlit
Other
980 stars 60 forks source link

Updated benchmarks as of May 2019 #140

Closed konung closed 5 years ago

konung commented 5 years ago

@k0kubun Thank you for a great gem.

Just FYI all major templating engines have had noticeable performance improvements ( of most relevance is an update to HAML 5.1 release a couple of days ago. ). And the benchmark on the front page is a bit old.

Hamlit still dominates thou, even surpassing erubi.

Here is my benchmark using 3 versions of Ruby && also trying out --jit with Ruby 2.7.0-dev


On Mac OSX 10.14.5 

ruby 2.5.3
ruby benchmark/slim/run-benchmarks.rb      
Warming up --------------------------------------
        erubi v1.8.0    13.142k i/100ms
         slim v4.0.1    12.643k i/100ms
         haml v5.1.0     8.206k i/100ms
         faml v0.8.1    11.386k i/100ms
       hamlit v2.9.2    13.440k i/100ms
Calculating -------------------------------------
        erubi v1.8.0    136.408k (± 2.2%) i/s -    683.384k in   5.012363s
         slim v4.0.1    134.288k (± 2.5%) i/s -    682.722k in   5.087319s
         haml v5.1.0     81.442k (± 6.1%) i/s -    410.300k in   5.056769s
         faml v0.8.1    105.163k (± 8.6%) i/s -    523.756k in   5.024666s
       hamlit v2.9.2    129.154k (± 5.4%) i/s -    645.120k in   5.010194s

Comparison:
        erubi v1.8.0:   136407.5 i/s
         slim v4.0.1:   134287.8 i/s - same-ish: difference falls within error
       hamlit v2.9.2:   129153.8 i/s - same-ish: difference falls within error
         faml v0.8.1:   105162.8 i/s - 1.30x  slower
         haml v5.1.0:    81442.2 i/s - 1.67x  slower

ruby 2.6.3
ruby benchmark/slim/run-benchmarks.rb                                                                                             1000ms  Thu May 23 17:55:53 2019
Warming up --------------------------------------
        erubi v1.8.0    21.104k i/100ms
         slim v4.0.1    20.078k i/100ms
         haml v5.1.0    11.572k i/100ms
         faml v0.8.1    18.249k i/100ms
       hamlit v2.9.3    21.737k i/100ms
Calculating -------------------------------------
        erubi v1.8.0    224.282k (± 2.2%) i/s -      1.140M in   5.083589s
         slim v4.0.1    213.624k (± 2.2%) i/s -      1.084M in   5.077896s
         haml v5.1.0    120.371k (± 1.8%) i/s -    601.744k in   5.000804s
         faml v0.8.1    193.789k (± 1.8%) i/s -    985.446k in   5.086803s
       hamlit v2.9.3    216.531k (± 6.7%) i/s -      1.087M in   5.043165s

Comparison:
        erubi v1.8.0:   224282.1 i/s
       hamlit v2.9.3:   216530.9 i/s - same-ish: difference falls within error
         slim v4.0.1:   213624.3 i/s - 1.05x  slower
         faml v0.8.1:   193789.3 i/s - 1.16x  slower
         haml v5.1.0:   120370.7 i/s - 1.86x  slower

ruby 2.7.0-dev
Warming up --------------------------------------
        erubi v1.8.0    20.948k i/100ms
         slim v4.0.1    19.311k i/100ms
         haml v5.1.0    10.293k i/100ms
         faml v0.8.1    16.617k i/100ms
       hamlit v2.9.3    21.214k i/100ms
Calculating -------------------------------------
        erubi v1.8.0    210.571k (± 6.7%) i/s -      1.068M in   5.099031s
         slim v4.0.1    196.450k (± 5.0%) i/s -    984.861k in   5.025960s
         haml v5.1.0    115.207k (± 4.0%) i/s -    576.408k in   5.011015s
         faml v0.8.1    175.592k (± 4.8%) i/s -    880.701k in   5.026984s
       hamlit v2.9.3    219.601k (± 5.2%) i/s -      1.103M in   5.037858s

Comparison:
       hamlit v2.9.3:   219601.1 i/s
        erubi v1.8.0:   210571.1 i/s - same-ish: difference falls within error
         slim v4.0.1:   196450.4 i/s - 1.12x  slower
         faml v0.8.1:   175592.5 i/s - 1.25x  slower
         haml v5.1.0:   115206.7 i/s - 1.91x  slower         

ruby 2.7.0-dev with --jit
ruby --jit benchmark/slim/run-benchmarks.rb                                                                                          37s  Thu May 23 18:00:41 2019
Warming up --------------------------------------
        erubi v1.8.0    17.880k i/100ms
         slim v4.0.1    17.615k i/100ms
         haml v5.1.0     9.759k i/100ms
         faml v0.8.1    15.093k i/100ms
       hamlit v2.9.3    16.615k i/100ms
Calculating -------------------------------------
        erubi v1.8.0    215.931k (± 9.0%) i/s -      1.073M in   5.023063s
         slim v4.0.1    210.368k (± 7.3%) i/s -      1.057M in   5.060784s
         haml v5.1.0    121.064k (± 3.4%) i/s -    605.058k in   5.003665s
         faml v0.8.1    189.276k (± 3.2%) i/s -    950.859k in   5.029025s
       hamlit v2.9.3    227.313k (± 3.8%) i/s -      1.146M in   5.051042s

Comparison:
       hamlit v2.9.3:   227312.6 i/s
        erubi v1.8.0:   215930.5 i/s - same-ish: difference falls within error
         slim v4.0.1:   210367.8 i/s - same-ish: difference falls within error
         faml v0.8.1:   189276.5 i/s - 1.20x  slower
         haml v5.1.0:   121063.9 i/s - 1.88x  slower
k0kubun commented 5 years ago

Thanks for reporting the information.

First, the benchmark is run with ruby 2.6 everyday on Travis, and the result is https://travis-ci.org/k0kubun/hamlit/jobs/536142677:

Comparison:
       hamlit v2.9.3:   191888.2 i/s
        erubi v1.8.0:   191440.1 i/s - 1.00x slower
         slim v4.0.1:   186977.8 i/s - 1.03x slower
         faml v0.8.1:   147069.5 i/s - 1.30x slower
         haml v5.1.0:   101107.0 i/s - 1.90x slower

I'm totally fine to regularly update these numbers but I don't think it has a significant difference from README right now. I don't think difference between Haml 5.0 and 5.1 has a significant impact on the generate code's performance either. Note that this benchmark is slightly naive for measurement noises.

Second, measuring the meaningful performance numbers with --jit is not that straight-forward. Your benchmark results with --jit would be mostly measuring overhead for having MJIT worker thread and a background compiler process. As all of them are in the same process, it could be even compiling another template engine's code while running a benchmark. The overhead itself is one problem of MJIT to be solved, but the numbers coming from it does not reflect the actual performance difference of native code generated by JIT. I'd need to modify the benchmark code for getting the meaningful results with --jit.

As I believe the current results are not that outdated, I'll consider doing those update stuffs when we publish Ruby 2.7. Thank you.