Our tests running against Browserstack sometimes falsely report all test passing.
While strictly speaking none of the tests failed, they certainly didn't all run.
For example,
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Individual Features failed with
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Full text of report below;
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
I need to be sure I am signed out for many of the auth steps to work properly ... OK
Sign in with incorrect email ... OK
Sign in with incorrect password ... OK
Standard Sign in ... OK
Sign up with Expired Token ... OK
Sign up with Expired Token that Expires after a number of days ... OK
Sign up with Expired Token that Expires after a specific date ... OK
Sign in with Disabled Account ... OK
Sign in with Expired Trial ... OK
I sign out ... OK
1 feature (1 passed)
10 scenarios (10 passed)
41 steps (41 passed)
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
========================================================================Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/lettuce_webdriver/parallel_runner.py", line 157, in worker_process
failfast=self.failfast))
File "/home/lex/local/lib/python2.7/site-packages/lettuce/core.py", line 1166, in run
call_hook('before_each', 'feature', self)
File "/home/lex/local/lib/python2.7/site-packages/lettuce/registry.py", line 88, in call_hook
callback(*args, **kw)
File "/home/lex/src/deus_lex/winslow/tests/features/terrain.py", line 40, in setup_browser
secret=world.browserstack_secret,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/home/lex/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
======================================================================================================================================================================================================================================================================================================================================================
Died with Message: u'5 sessions are currently being used. Please upgrade to add more parallel sessions.'
0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
View judges for law firm ... OK
View judges for attorney ... OK
1 feature (1 passed)
2 scenarios (2 passed)
13 steps (13 passed)
Ensure Each District Court Page Renders ...
1 feature (1 passed)
92 scenarios (92 passed)
184 steps (184 passed)
Our tests running against Browserstack sometimes falsely report all test passing. While strictly speaking none of the tests failed, they certainly didn't all run.
For example,
Individual Features failed with
Full text of report below;