Raku / whateverable

🤖 Different IRC bots that operate on a bunch of prebuilt Rakudo versions
https://gist.github.com/Whateverable
GNU Affero General Public License v3.0
18 stars 14 forks source link

Automatically detect wrong bisection results due to flapping code #378

Open AlexDaniel opened 4 years ago

AlexDaniel commented 4 years ago

Currently, there's this check: https://github.com/Raku/whateverable/blob/cf9bcd03b154bcc570aca8a2628099a408cde460/xbin/Bisectable.p6#L266-L268

However, @lizmat++ and @Altai-man++ came up with an idea of doing this check automatically by comparing .moarvm files of the final commit and its parent.

I have a suspicion that rakudo in the past did not produce the same .moarvm file reproducibly, meaning that the check won't work on old commits. However, we can still do some checks, for example we can check if all changed files in the commit were to doc files.

AlexDaniel commented 4 years ago

\<nine> No, .moarvm files won't be identical since the compiler's id contains the SHA of the full source code

Bummer. So there's no relatively easy way of detecting that. I'll keep this open in case somebody has a better idea on how to achieve this.

Note that the commit that prompted this discussion was https://github.com/rakudo/rakudo/commit/fb6a12a, which is not just a README or doc change, so it's not very easy to figure out that it didn't really change any functionality.

IRC discussion: https://colabti.org/irclogger/irclogger_log/raku-dev?date=2020-06-27#l53