Closed imjoehaines closed 2 years ago
I believe I've copied the exact Ruby/framework version matrices over from Buildkite, but it'd be good to double check as there are a lot of combinations!
I think I saw a couple of instances where you end up testing a couple of extra combination that were not previously covered (but it made sense to as it made the matrix simply and "why not" test those extra combos.
it's probably worth reviewing if we really need to run this many combinations of Ruby & framework versions as it's unlikely that something will fail specifically with e.g. Rails 5 on Ruby 2.4, but not any other Ruby version. For now, though, I think it makes sense to keep the versions the same so that this is a 1:1 lift of Buildkite → GH Actions
Yes, agreed on both points - try to make this a lift-and-shift and now, but consider dropping some intermediate versions. E.g. if something works on Ruby 2.0 and 2.7, is it likely to break anywhere in between? (It might of course if there are lots of conditionals in the code!).
Goal
This PR moves the Maze Runner tests onto GH Actions
This uses a "reusable workflow" defined in
run-maze-runner.yml
. Essentially this allows us to define how to run Maze Runner separately to what to run with Maze Runner. This is useful because we have a lot of separate matrices for various frameworks, e.g. our Rails 7 tests only run against Ruby 2.7, 3.0 & 3.1 but our "plain" tests run against every version of Ruby from 1.9I believe I've copied the exact Ruby/framework version matrices over from Buildkite, but it'd be good to double check as there are a lot of combinations![^1]
I've not limited the concurrency at all as (in my limited testing) it was much faster to run all of the builds at once and let GH automatically limit things, rather than trying to manipulate it into prioritising the slower tests (Rails & plain ruby)
Overall this gives a huge speedup in CI time — from ~30-60 minutes on Buildkite to ~15 minutes on GH Actions (which is only that long because our Rails 6 on Ruby 2.5 test takes ~14 minutes alone)
[^1]: it's probably worth reviewing if we really need to run this many combinations of Ruby & framework versions as it's unlikely that something will fail specifically with e.g. Rails 5 on Ruby 2.4, but not any other Ruby version. For now, though, I think it makes sense to keep the versions the same so that this is a 1:1 lift of Buildkite → GH Actions