bugsnag / bugsnag-ruby

BugSnag error monitoring & reporting software for rails, sinatra, rack and ruby
https://docs.bugsnag.com/platforms/ruby
MIT License
246 stars 174 forks source link

Move Maze Runner tests to GH Actions #737

Closed imjoehaines closed 1 year ago

imjoehaines commented 1 year ago

Goal

This PR moves the Maze Runner tests onto GH Actions

This uses a "reusable workflow" defined in run-maze-runner.yml. Essentially this allows us to define how to run Maze Runner separately to what to run with Maze Runner. This is useful because we have a lot of separate matrices for various frameworks, e.g. our Rails 7 tests only run against Ruby 2.7, 3.0 & 3.1 but our "plain" tests run against every version of Ruby from 1.9

I believe I've copied the exact Ruby/framework version matrices over from Buildkite, but it'd be good to double check as there are a lot of combinations![^1]

I've not limited the concurrency at all as (in my limited testing) it was much faster to run all of the builds at once and let GH automatically limit things, rather than trying to manipulate it into prioritising the slower tests (Rails & plain ruby)

Overall this gives a huge speedup in CI time — from ~30-60 minutes on Buildkite to ~15 minutes on GH Actions (which is only that long because our Rails 6 on Ruby 2.5 test takes ~14 minutes alone)

[^1]: it's probably worth reviewing if we really need to run this many combinations of Ruby & framework versions as it's unlikely that something will fail specifically with e.g. Rails 5 on Ruby 2.4, but not any other Ruby version. For now, though, I think it makes sense to keep the versions the same so that this is a 1:1 lift of Buildkite → GH Actions

twometresteve commented 1 year ago

I believe I've copied the exact Ruby/framework version matrices over from Buildkite, but it'd be good to double check as there are a lot of combinations!

I think I saw a couple of instances where you end up testing a couple of extra combination that were not previously covered (but it made sense to as it made the matrix simply and "why not" test those extra combos.

twometresteve commented 1 year ago

it's probably worth reviewing if we really need to run this many combinations of Ruby & framework versions as it's unlikely that something will fail specifically with e.g. Rails 5 on Ruby 2.4, but not any other Ruby version. For now, though, I think it makes sense to keep the versions the same so that this is a 1:1 lift of Buildkite → GH Actions

Yes, agreed on both points - try to make this a lift-and-shift and now, but consider dropping some intermediate versions. E.g. if something works on Ruby 2.0 and 2.7, is it likely to break anywhere in between? (It might of course if there are lots of conditionals in the code!).