Open lanceblais opened 9 years ago
Have you tried running in parallel by using the Rake task?
bundle exec rake sauce:spec
Is the parallelization at the describe block level, the file level, or other?
parallelization is performed using parallel_tests which runs one process per file.
@bootstraponline we're using rspec with a spec_helper. We aren't using a Rakefile. Any help on why we'd use that?
It would be interesting to know if it worked via Rakefile or if the error remained the same.
Ah I see. Ok, I assume the details are on the page about setting up the Rakefile. Our QA Engineer just always ran rspec :-) -- I'll get back to you
I have an example repo.
require 'bundler/setup'
require 'sauce'
bundle exec rake sauce:spec
The Sauce gem patches parallal_test's test division like so:
# psuedocode
files = ['a1', 'a2', 'b1', 'b2', 'd1', 'd2']
thread_1 = ['a1',d1']
thread_2 = ['a2','d2']
thread_3 = ['b1']
thread_4 = ['b2']
By doing tests in parallel in this manner, we have a greater chance of evening out test time. The default Sauce gem running pattern, that of running each test sequentially against each browser, doesn't parallelise as well; A long running test will run on every browser in turn in the same thread, potentially taking significantly longer then every other thread. Additionally, in instances where there is more available concurrency then tests, there's no way to use higher concurrency.
Unfortunately, running in this test-distributing fashion requires us to do some funky stuff to load the required test config before a test run, hence relying on a rake
task. The gem complains about not finding any browser because it knows it ran from the parallel_test integration, but not that it didn't run from the Sauce task, so it thinks something has gone wrong.
It'd be possible to fix this, I guess, by setting an environment variable when the rake task runs, telling child tasks that they're running from the Sauce specific integration; If we were to do that, however, I'd still want to issue a message saying that parallelism of that kind isn't always as efficient.
parallel_tests uses processes to run in parallel not threads (even though it waits for them to finish via threads). In the ideal situation, it'd be per test and not per file via parallel_split_test. Manually breaking large test files down into smaller files for better performance isn't fun.
@bootstraponline How do I make the Rakefile only run tests within a certain folder? We've got 2 old ones and a new one that we run and running bundle exec rake sauce:spec
is looking in the other 2 folders.
The tests get kicked off on Saucelabs by running bundle exec rake spec
but as soon as I add the sauce
part to the command it fails by trying to look in other directories.
So: same issue as above.
require 'bundler/setup'
require 'sauce'
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new(:spec) do |t|
t.pattern = 'spec/features/end_to_end/**/*.rb'
end
@bootstraponline How do I make the Rakefile only run tests within a certain folder?
@lanceblais The sauce:spec rake task accepts options.
rake sauce:spec[files,concurrency,test_options,parallel_options] # Run specs in parallel on Sauce Labs
Try passing the target folder as the files
option to restrict execution.
@bootstraponline
I was only able to give it one file at a time (unless there's some way to do regex) by running: bundle exec rake sauce:spec test_files="spec/features/end_to_end/1/ma_agenda/ma_fixes.rb
When I run this command I get exactly the same output as I did when running with parrallel_rspec
above.
So to answer your question: rake sauce:spec
and parallel_rspec
are now giving me the same behaviour of no browser has been configured.
.
Sorry for the delay, now what?
I recommend creating a reduced test case that reproduces the issue and uploading it to github. The example I have on GitHub is working fine so I think there's something unique about your configuration.
Hello,
We're using RSpec and SauceLabs to run our Selenium tests.
We've followed all the information on the Wiki and we're able to run
bundle exec rspec spec/path/to/e2e
.When we enable parallelization and run
bundle exec parallel_rspec spec/path/to/e2e
we get this error:Note: launch_cms_homepage() is a method defined in one of the helpers we pull in.
Has this been seen before?
A related question, what strategy does the parallelization take for a folder of specs?
The
e2e
folder hase2e/1/test1.rb
which has a bunch of describe blocks ande2e/1/test2.rb
also has a bunch of specs. Is the parallelization at the describe block level, the file level, or other?Thank you!