search

fastruby / fast-ruby

:dash: Writing Fast Ruby :heart_eyes: -- Collect Common Ruby idioms.
https://github.com/fastruby/fast-ruby
5.68k stars 377 forks source link

[FEATURE REQUEST] Use the project's WIKI to share results for each Ruby version with latest CI results #206

Open arielj opened 2 years ago

arielj commented 2 years ago

Currently, the results are in the README file, but all numbers are from one specific ruby version. Trying to add information for more rubies can make the README a mess and super large to follow.

Another source for the benchmark results is the result of CI runs which shows up-to-date and more rubies, but it's not formatted for readability, it's just the output of the rake command which is not ideal.

Maybe another option is to use the Wiki of the project creating one page with the results for each Ruby that's tested, and then use a github action to update the results there so each page is always automatically updated with the latest results from CI.

I've never done this but I think it's possible, there's this action for example that creates a Wiki page given a markdown file https://github.com/Decathlon/wiki-page-creator-action. So I understand it's possible to create/edit Wiki pages if we can output the benchmark results as a .md file using the github API.

This would help finding benchmarks relevant to the Ruby version the user is using, since different ruby versions may add new optimizations to some of the code variants and old Ruby benchmarks would be inaccurate.

Joshfindit commented 2 years ago

I like the core concepts here, and think there might be an easier option:

Store the results as .md (and json/yaml) files in the filesystem, with the folder as the Ruby version number.

That would give the following benefits:

  1. The wiki could canonically link to the folder
  2. Since it could be stored as results/version/README.md, github would display the rendered markdown when users navigate through
  3. CI would only need to commit the relevant files
  4. Users could create tools to track differences between Ruby versions
etagwerker commented 2 years ago

We might be able to leverage benchmark.fyi which is an OSS Rails app that stores benchmark-ips results.

I've submitted a PR that shares the results to https://ips.fastruby.io (powered by benchmark.fyi) -- you can see it here: https://github.com/fastruby/fast-ruby/pull/203

One enhancement that I'd like to add to benchmark-ips and benchmark.fyi is more contextual information:

Not sure how all of this would tie together, but maybe there could be a wiki page that keeps track of the benchmarks we run and share for this repo?