nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.8k stars 1.31k forks source link

[2.0.2] test_workers do not correctly spawn drivers #3028

Open chrisvanrun opened 2 years ago

chrisvanrun commented 2 years ago

Describe the bug

Starting with enabled test_workers seems to not use selenium-webdriver correctly. When running a test session with them enabled, the parallel tests do not seem to correctly want to spawn chromedrivers (only the first worker gets one spawned):

Starting ChromeDriver on port 9515...
test.js ⚠ Failed to connect to ChromeDriver on localhost with port 9515.

Whereas adding --parallel to the CLI call seems to fix it. Notably, in verbose the tests report (using --parallel):

Starting ChromeDriver on auto-generated port...

The strange thing is that there seems to be some race condition when only test_workers are enabled. Making this extremely flaky. In some situations it passes checks (likely connects to the default port), one or more tests start running but it then has a ECONNRESET (socket hang up) when the first driver process stops: but only sometimes. Sometimes the driver is kept alive between starting and stopping test suites: super weird. Setting parallel_process_delay makes it less flaky and makes it always fail.

I did not add an example script because this seems independent on test-suite contents.

Configuration

nightwatch.conf.js (with `test_workers` enabled)

```js module.exports = { 'src_folders' : ['nightwatch/tests'], 'page_objects_path': 'nightwatch/pages', 'output_timestamp': true, 'live_output': true, 'webdriver' : { 'start_process': true, }, 'test_settings' : { 'default' : { 'launch_url': 'http://localhost:8888/cirrus', 'desiredCapabilities': { 'browserName': 'chrome', 'goog:chromeOptions': { 'w3c': true, }, 'globals': { 'waitForConditionTimeout': 10000, }, }, 'screenshots': { 'enabled': false, 'path': './screenshots', 'on_error': true, 'on_failure': true, }, }, }, // Parallel settings 'parallel_process_delay': 200, 'test_workers': { 'enabled': true, 'workers': 4, }, } ```

nightwatch.conf.js (with `test_workers` disabled)

```js module.exports = { 'src_folders' : ['nightwatch/tests'], 'page_objects_path': 'nightwatch/pages', 'output_timestamp': true, 'live_output': true, 'webdriver' : { 'start_process': true, }, 'test_settings' : { 'default' : { 'launch_url': 'http://localhost:8888/cirrus', 'desiredCapabilities': { 'browserName': 'chrome', 'goog:chromeOptions': { 'w3c': true, }, 'globals': { 'waitForConditionTimeout': 10000, }, }, 'screenshots': { 'enabled': false, 'path': './screenshots', 'on_error': true, 'on_failure': true, }, }, }, // Parallel settings 'parallel_process_delay': 200, 'test_workers': { 'enabled': false, 'workers': 4, }, } ```

Your Environment

Executable Version
nightwatch --version 2.0.2
npm --version 8.1.2
yarn --version 1.22.15
node --version v16.13.2
Browser driver Version
chromedriver 97.0.4692.71
OS Version
Windows 11 22534
WSL2 Ubuntu 20.04
mostmentor commented 2 years ago

I have same issue while trying to run on AWS Device Farm.

AutomatedTester commented 1 year ago

@swrdfish is this now fixed?