Enable re-run feature.

[mariozaizar]

Ok, we love parallel_test gem. But also we're getting some random failures with it (seems like some legacy tests have hard dependencies, what a shame).

Anyway, we will love if we are be able to re-run those failures at least one more time in isolation, and then see if we need to mark it as failed or not.

So, for example:

bundle exec rake parallel:spec -o failures.txt;

So, we could do:

bundle exec rake parallel:spec -i failures.txt;

[grosser]

rake takes no hostages (or arguments). making this work for different test frameworks could be complicated. you can use the rpsec failurelogger and then format it and use it as input for parallel_rspec <file1> <file2>, you could make this into a rerunlogger that only lists unique file names, which would simplify the process

[KurtPreston]

So, I've gotten this working with the following (relatively ugly) approach:

Create a new rake task lib/tasks/parallel_rerun.rake

namespace :parallel do
  namespace :features do
    desc "Run cucumber tests, re-running any failures"
    task :rerun, :cores do |t, args|
      cores = args[:cores]
      command = "bundle exec rake parallel:features[#{cores}]"
      puts "*********** RUNNING PARALLEL BUILD *********"
      puts "***** #{command} *****"
      unless system(command)
        # If parallel build fails, try executing failing tests in serial
        puts "*********** PARALLEL BUILD FAILED ************"
        puts "************ re-running failures *************"
        unless system('bundle exec cucumber --profile rerun_failures')
          fail "Rerun failed"
        end
      end
    end
  end
end

I then modified config/cucumber.yml to support multiple parallel outputs as follows:

<%

rerun_text = ''
# Concatanate all rerun test file output
(1..8).each do |test_env_num|
  env_num_str = if (test_env_num == 1)
    ''
  else
    test_env_num.to_s
  end

  rerun_file = "tmp/rerun#{env_num_str}.txt"
  if File.file?(rerun_file)
    rerun_text << ' ' + IO.read(rerun_file)
    File.delete(rerun_file)
  end
end

rerun_failures_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} #{rerun_text}"

rerun_opts = rerun_text.strip.empty? ? "--format #{ENV['CUCUMBER_FORMAT'] || 'progress'} features" : rerun_failures_opts

std_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} --strict --tags ~@wip"

%>

default: <%= std_opts %> features
wip: --tags @wip:3 --wip features
rerun: <%= rerun_opts %> --format rerun --out tmp/rerun.txt --strict --tags ~@wip
rerun_failures: <%= rerun_failures_opts %>
parallel: --strict --format rerun --out tmp/rerun<%= ENV['TEST_ENV_NUMBER'] %>.txt --format pretty

After all this, I get the ability to run parallel cukes, while automatically re-running any failures in serial using the following command:

rake parallel:features:rerun

[grosser]

is --format rerun an official cucumber thingy ?

[grosser]

if you move the parsing logic into the rake task would it still work <-> would be more reusable then telling people to past big code-blocks into their cucumber.yml

[KurtPreston]

--format rerun is a standard cucumber profile, unfortunately it doesn't natively work with parallel_tests. I'll look into your suggestion about pulling out the logic into the Rake task. That's probably a much cleaner way to do it. There will have to be some changes to config/cucumber.yml, however. In particular, you need to tell it to create a separate re-run file per thread.

[justin808]

If you're using rspec, you should consider using

  gem 'rspec-retry'

I have some capybara tests that seem to fail randomly. This addresses the issue.

[simonoff]

Any updates? @KurtPreston provided most usable hack?

[smartkiwi]

This merged pull request provides implements solution to @KurtPreston suggestion - sends rerun details into single file using custom formatter. https://github.com/grosser/parallel_tests/pull/202

[bootstraponline]

Do any of the rspec retry gems work with parallel_tests?

• rspec-rerun
• rspec-retry

I think I'll have to add a custom runner to parallel tests.

[grosser]

You should be able to get something to work if it prints out a list of failed files and then rerun these files ... you could possibly also run multiple examples with foospec.rb:12:45:33

[carolmirakove]

@bootstraponline I use rspec-retry with parallel_tests, it works perfectly.

[bootstraponline]

@bootstraponline I use rspec-retry with parallel_tests, it works perfectly.

Thanks for the tip. I'm seeing ERROR Job is not in progress issues when using rspec-retry with Sauce Labs. I'll look into why it's apparently not triggering the rspec before/after on test retry.

[carolmirakove]

here's my spec_helper.rb in case it's useful: https://gist.github.com/carolmirakove/1275cadb08ee1cf89618

I didn't change anything in my Rakefile when I added rspec-retry into the mix.

[bootstraponline]

Thanks! It looks like the only difference is I'm using the sauce_ruby gem which has a bunch of known problems.

[dylanlive]

I'm using rspec_rerun and am about to implement parallel_tests.

I'll do some digging to see if I can get rspec_rerun to work with this, although I might just switch over to rspec_retry. If anyone has had any luck with rspec_rerun, would appreciate your knowledge!

Thanks for pointing me to rspec_retry, @carolmirakove & @justin808

[bootstraponline]

I had a bug in my code (helper method using old driver) and once fixed then rspec-retry worked perfectly.

rspec_rerun seems to have less features (unable to specify per test retry values) and uses ENV vars instead of an rspec config.

I think the solution to this issue would be to add a note in the readme about rspec-retry.

[grosser]

thanks for the PR ! :)

[OpakAlex]

how i can rerurn failed specs? any ideas? not cucumbers ;)

[grosser]

use rspec-retry ? otherwise use an additional formatter to send a copy of the test output to another file and then rerun all the tests in that file

[ezaki-viv]

This did work for me really well. Try using an environment variable named SPEC_OPTS to pass the argument '--only-failures' to parallel_rspec.

SPEC_OPTS='--only-failures' parallel_rspec -n 4 Spec/*.rb

[VikasKumar190]

Hi @ezaki-viv , even after doing the following the status file is not getting saved at the specified location.

RSpec.configure do |c|
  c.example_status_persistence_file_path = "examples.txt"
end

Did you also encounter any such problem?

[ezaki-viv]

Hi @ezaki-viv , even after doing the following the status file is not getting saved at the specified location.

RSpec.configure do |c|
  c.example_status_persistence_file_path = "examples.txt"
end

Did you also encounter any such problem?

I could see no mistake in the assignment. Ideally, it should save. Can I have a look into your spec_helper file ?

[VikasKumar190]

Hey! @ezaki-viv , I found out that it was being saving into spec/log/meta/example.txt The problem that I am seeing now is that when I run my tests with SPEC_OPTS='--only-failures' it shows me the following output:

Run options: include {:last_run_status=>"failed"}

All examples were filtered out

This is the exact command that I am running

SPEC_OPTS='--only-failures' bundle exec parallel_rspec -n 2 --only-group 2 spec/features/*.rb

[ezaki-viv]

Hey! @ezaki-viv , I found out that it was being saving into spec/log/meta/example.txt The problem that I am seeing now is that when I run my tests with SPEC_OPTS='--only-failures' it shows me the following output:

Run options: include {:last_run_status=>"failed"}

All examples were filtered out

This is the exact command that I am running

SPEC_OPTS='--only-failures' bundle exec parallel_rspec -n 2 --only-group 2 spec/features/*.rb

This is because your examples.txt has no entries in it.

[VikasKumar190]

It has entries in it

[ezaki-viv]

It has entries in it

Are u sure that the examples.txt file is accessible to your project then?

[VikasKumar190]

Since the file was getting written with the failures, I assumed that it is accessible while re-running but that wasn't the case. @ezaki-viv , Thanks for pointing it out.

[bsa7]

@bootstraponline I use rspec-retry with parallel_tests, it works perfectly.

rspec-retry Work fine with:

  * parallel_tests (3.5.2)
  * rspec (3.9.0)
  * rails (6.1.3)