[REQUEST] Support parallel testing

[mrleeio]

Description

Splitting test files into multiple processes on continuous integration platforms has become common to reduce run time.

For example, on GitHub Actions, you might create a workflow which utilizes a matrix strategy. You can read a blog post on the subject here.

The compare mode for next_rails is too simplistic to account for this type of setup. Each process will create a different shitlist file which fails the diff comparison.

Possible Implementation

Create a new comparison mode that checks file content.

Committed file:

  "./spec/file_one.rb": [
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ...",
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ..."
  ],
  "./spec/file_two.rb": [
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ...",
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ..."
  ]

Matrix Job One:

  "./spec/file_one.rb": [
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ...",
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ..."
  ]

Matrix Job Two:

  "./spec/file_two.rb": [
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ...",
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ..."
  ]

Both Matrix jobs fail when comparing using a diff. However, it would pass if we searched for that content in the committed file.

Resources:

• Matrix strategy
• Parallel GitHub Actions for RSpec

I will abide by the code of conduct

[etagwerker]

@mrleeio Thanks for submitting this issue!

I'm curious, how are you running the diff command? Is it running in CI or in your dev environment?

For a similar problem (parallelized jobs in CI) with SimpleCov data, we usually recommend merging SimpleCov files from all the different artifacts generated in CI and then using them.

I'm not sure what's the best solution, but maybe doing a merge of the JSON files might be necessary...

[mrleeio]

@etagwerker sorry I may have been unclear. I only use the default comparison mode.

DEPRECATION_TRACKER=compare rspec

It will always fail when this is run on separate processes being compared against a single shitlist file. Spec distribution among processes is completely random.

This command works wonderfully locally or when specs are run within a single process.