search

buildkite / test-splitter

go test splitting client
MIT License
6 stars 1 forks source link

[Question] How to run this with rspec? #118

Open brett-anderson opened 1 month ago

brett-anderson commented 1 month ago

I am evaluating Buildkite as a replacement for our CI and I came across this repository. We currently use https://github.com/grosser/parallel_tests to split our rspec suite by runtime.

If we were to move to Buildkite, it would be nice to use the Test Analytics, as well as some sort of tool to partition the test suite.

Can you provide examples of how one would use this with an rspec test suite?

Thank you!

wooly commented 1 month ago

Hi @brett-anderson, thanks for reaching out!

We're currently working on some public documentation for integrating the test splitter into your pipeline, but for now I can describe here what this looks like (the documentation we do have is in-tool).

The steps would be as follows:

  1. Add the test splitter binary from the Github releases to your buildkite agents and make available in docker if you're containerizing your tests.
  2. Configure test-collector-ruby library to collect span data for your test suite. If you're looking at using Test Analytics, then you'd likely be doing this anyway.
  3. Configure your pipeline step to use the test splitter instead of directly calling RSpec. Something like the instructions here: https://github.com/buildkite/test-splitter?tab=readme-ov-file#run-the-test-splitter

There are a few differences between the Buildkite test splitter and Parallel Tests that might be worth calling out:

  1. Test Splitter doesn't use a parallel database approach, since the partitions will be running on different nodes with no access to a shared database. It's effectively running rspec with a subset of your tests, where the union of those subsets is the whole suite.
  2. Our plan is to support multiple test runners like Cypress and Jest in the future, so deep integration with runners is kept to a minimum and the test splitter serves as an orchestrator around the runner (in this case RSpec). This means no gems or packages to install.

We'd really appreciate some feedback on the setup process, so if you have a Buildkite account and want to have a crack at integrating the test splitter into your pipeline, let me know.