Support for parallel tests

[jarommadsen]

Loving the tool! The only thing I want for is to be able to run them in parallel.

You mention in the README that rspec_n can run tests in parallel by using the -c option without showing an example but I don't think this covers what I'm after.

I've installed the parallel_rspec gem which can run tests in parallel but what I want to do is have each retry to use a different worker.

For example: I create 4 workers in parallel_rspec and run rspec_n on a specific spec file. I want 4 of those attempts to use an individual worker simultaneously to cut the total runtime in half.

Is there a workaround that I'm not seeing?

[roberts1000]

Hi @jarommadsen. It's nice to hear you're enjoying the tool! The Use a Custom Command to Start RSpec has some examples on how to use the -c option. It might give you an idea on how to use it to run parallel specs, but if I'm understanding your use case right, rspec_n probably can't support the kind of parallelization you described.

rspec_n places all of the burden for running rspec on the rspec gem, parallel_rspec or some other library. It just repeats the same command n times, captures the result of each "run", and ensures each one starts in the same way (and with the same environment). parallel_rspec is responsible for parallelizing spec examples and managing workers. All of the configuration for parallelization should happen directly in that gem. If parallel_rspec doesn't support it, it's probably out of scope for rspec_n.

If we add some parallelization support in rspec_n, I suspect it would be limited to making rspec_n smartly start rspec when it sees a gem like parallel_rspec is present (so folks don't have to use the -c option themselves).

[jarommadsen]

@roberts1000 I think I wasn't clear enough in my original post but the functionality I'm after is to reduce the amount of time running the same test multiple times is taking.

Specifically, I'm using rspec_n to assist in investigating flaky tests. I'm trying to get an idea of how often a specific test flakes and so I'm running that single test 100 times in a row.

rspec_n is great for this use case but will only run each attempt on a single thread/worker. I'd like to be able to run some of those attempts in parallel like specifying a flag like rspec_n 100 --workers=8 path/to/spec/file.

Any chance this could be implemented?

[roberts1000]

@jarommadsen Thanks for the extra info. That helps me understand better. If there's a way to pass info down to parallel_rspec, and let parallel_rspec handle it from there, that doesn't sound like it would be too tough to implement. I haven't used parallel_rspec myself yet, so I'm not sure if that could work, but from your earlier comments, it doesn't sound like there's an obvious way to do it. If you see a way to do that, I could potentially add support for it.

It would be more difficult to add the concept of workers directly intorspec_n so it can run multiple instances of an app's test suite in parallel. Even if it's just one, or a few files, that are getting targeted, an app's test environment would still get loaded multiple times concurrently, and rspec_n would have to manage them across a variable number of workers. I suspect it would require a major rewrite of the internals. I don't have the bandwidth available to work on it, but I'm open to a PR if someone would like to take it on.

This part is just me "thinking-out-loud" a bit. I'm interested if you see it the same way... If someone is running a full test suite, that's already parallelized, and they have more tests/files than workers, parallel_rspec is already going to take advantage of the available workers to run tests concurrently. Having rspec_n add a second level of parallelization doesn't seem beneficial in that case because the workers are already fully utilized. The benefit wouldn't happen until the number of tests/files is less than the number of workers. At that point, a tool that can parallelize multiple test suites could take advantage of free workers.