webdriverio / webdriverio

Next-gen browser and mobile automation test framework for Node.js
http://webdriver.io
MIT License
8.97k stars 2.49k forks source link

[🐛 Bug]: BrowserStack Parallel runs issues for v9 #13425

Open daemontoxa opened 3 weeks ago

daemontoxa commented 3 weeks ago

Have you read the Contributing Guidelines on issues?

WebdriverIO Version

latest

Node.js Version

v20.9.0

Mode

WDIO Testrunner

Which capabilities are you using?

capabilities: {
            myAppiumDriver: {
                capabilities: {
                    platformName,
                    'appium:platformVersion': process.env.OS_VERSION,
                    'appium:deviceName': process.env.DEVICE,
                    'appium:app': process.env.BROWSERSTACK_APP_ID,
                    'appium:optionalIntentArguments': '--ez autoBindDevice true',
                    'appium:processArguments': '{ "args": ["-autoBindDevice", "true"] }',
                    'appium:automationName': automationName,
                    'appium:build': `${ENV} ${platformName} ${bitriseBuildNumber}`,
                    'appium:projectName': `${appName} WDIO Automation`,
                    'appium:noReset': false,
                    'appium:autoGrantPermissions': true, // Android only
                    'bstack:options': {
                        appiumVersion: '2.6.0',
                        buildName: `${ENV} ${platformName} ${bitriseBuildNumber} ${environments.LANG}`,
                        userName: process.env.BROWSERSTACK_USERNAME,
                        accessKey: process.env.BROWSERSTACK_ACCESS_KEY,
                        enableBiometric: true,
                        idleTimeout: '4000',
                    },
                },
            },
        },

What happened?

We using parallel runs in BrowserStack real devices (using 'maxInstances' setting) We use the BDD approach and have many Feature files with testing scenarios

After updating to WDIO v9 I've noticed that not all tests were completed during parallel runs For example, we have 10 Feature files with scenarios and set 'maxInstances: 6' - and as a result, we got only results of the first 6 feature scenarios assigned on free 6 BS parallel devices, and all other attempts to run 4 Feature files which are left are failed with the following error in logs:

2024-08-19T08:54:29.897Z INFO @wdio/local-runner: Run worker command: run 2024-08-19T08:54:33.694Z DEBUG @wdio/runner: init multiremote session 2024-08-19T08:54:33.767Z ERROR @wdio/runner: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/protocol-stub.js' imported from /builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js at new NodeError (node:internal/errors:406:5) at finalizeResolution (node:internal/modules/esm/resolve:233:11) at moduleResolve (node:internal/modules/esm/resolve:845:10) at defaultResolve (node:internal/modules/esm/resolve:1043:11) at nextResolve (node:internal/modules/esm/hooks:833:28) at resolveBase (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:3126) at async resolveDirectory (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:3492) at async resolve (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:4355) at async nextResolve (node:internal/modules/esm/hooks:833:22) at async Hooks.resolve (node:internal/modules/esm/hooks:278:24)

With WDIO v8 this problem is not reproduced

What is your expected behavior?

All tests should be successfully executed during parallel runs using BrowserStack devices

How to reproduce the bug.

It's hard to create a reproducible example of this issue because BrowserStack requires setting up personal tokens to trigger runs on their side I tried to add as much as possible info inside What's happened section

Relevant log output

2024-08-19T08:54:29.897Z INFO @wdio/local-runner: Run worker command: run
2024-08-19T08:54:33.694Z DEBUG @wdio/runner: init multiremote session
2024-08-19T08:54:33.767Z ERROR @wdio/runner: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/protocol-stub.js' imported from /builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js
    at new NodeError (node:internal/errors:406:5)
    at finalizeResolution (node:internal/modules/esm/resolve:233:11)
    at moduleResolve (node:internal/modules/esm/resolve:845:10)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at nextResolve (node:internal/modules/esm/hooks:833:28)
    at resolveBase (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:3126)
    at async resolveDirectory (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:3492)
    at async resolve (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/tsx/dist/esm/index.mjs?1724057651811:2:4355)
    at async nextResolve (node:internal/modules/esm/hooks:833:22)
    at async Hooks.resolve (node:internal/modules/esm/hooks:278:24)

Code of Conduct

Is there an existing issue for this?

daemontoxa commented 3 weeks ago

Also got the following error log today

2024-08-20T12:01:43.604Z INFO @wdio/local-runner: Run worker command: run 2024-08-20T12:01:44.626Z DEBUG @wdio/runner: init multiremote session 2024-08-20T12:01:44.637Z DEBUG @wdio/utils:initializeServices: initialize service "browserstack" as NPM package 2024-08-20T12:01:44.951Z DEBUG @wdio/utils:shim: Finished to run "beforeSession" hook in 0ms 2024-08-20T12:01:45.863Z DEBUG @wdio/runner: init multiremote session 2024-08-20T12:01:45.872Z INFO webdriver: Initiate new session using the WebDriver protocol 2024-08-20T12:01:45.872Z INFO @wdio/utils: Connecting to existing driver at https://hub-cloud.browserstack.com:443/wd/hub 2024-08-20T12:01:45.876Z INFO webdriver: [POST] https://hub-cloud.browserstack.com/wd/hub/session 2024-08-20T12:01:45.876Z INFO webdriver: DATA { capabilities: { alwaysMatch: { platformName: 'ANDROID', 'appium:platformVersion': '13.0', 'appium:deviceName': 'Google Pixel 7 Pro', 'appium:app': 'bs://f8aee7733c32d6e7f5ae75c1d0f4feafceac22a1', 'appium:optionalIntentArguments': '--ez autoBindDevice true', 'appium:processArguments': '{ "args": ["-autoBindDevice", "true"] }', 'appium:automationName': 'UiAutomator2', 'appium:build': 'ALPHA ANDROID 590780234', 'appium:projectName': 'NOMO WDIO Automation', 'appium:noReset': false, 'appium:autoGrantPermissions': true, 'bstack:options': [Object] }, firstMatch: [ {} ] }, desiredCapabilities: { platformName: 'ANDROID', 'appium:platformVersion': '13.0', 'appium:deviceName': 'Google Pixel 7 Pro', 'appium:app': 'bs://f8aee7733c32d6e7f5ae75c1d0f4feafceac22a1', 'appium:optionalIntentArguments': '--ez autoBindDevice true', 'appium:processArguments': '{ "args": ["-autoBindDevice", "true"] }', 'appium:automationName': 'UiAutomator2', 'appium:build': 'ALPHA ANDROID 590780234', 'appium:projectName': 'NOMO WDIO Automation', 'appium:noReset': false, 'appium:autoGrantPermissions': true, 'bstack:options': { appiumVersion: '2.6.0', buildName: 'ALPHA ANDROID 590780234 EN', userName: '*', accessKey: '**', enableBiometric: true, idleTimeout: '4000', wdioService: '9.0.4', buildIdentifier: '590780234' } } } 2024-08-20T12:03:45.934Z ERROR webdriver: Error: Failed to fetch https://hub-cloud.browserstack.com/wd/hub/session: [object Object] at FetchRequest._libRequest (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:1376:15) at FetchRequest._request (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:1283:20) at startWebDriverSession (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:898:16) at Function.newSession (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:1511:41) at remote (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:6964:20) at (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:7008:24) at async Promise.all (index 0) at multiremote (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:7006:3) at async initializeInstance (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:82:20) at async Runner._startSession (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:778:23) 2024-08-20T12:03:46.112Z ERROR @wdio/runner: Error: Failed to create session. Failed to fetch https://hub-cloud.browserstack.com/wd/hub/session: [object Object] at startWebDriverSession (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:902:11) at Function.newSession (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriver/build/index.js:1511:41) at remote (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:6964:20) at (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:7008:24) at async Promise.all (index 0) at multiremote (/builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/webdriverio/build/index.js:7006:3) at async initializeInstance (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:82:20) at async Runner._startSession (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:778:23) at async Runner._initSession (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:752:22) at async Runner.run (file:///builds/bb2-bank/qa-testing/nomo-e2e-testing/node_modules/@wdio/runner/build/index.js:659:16) 2024-08-20T12:03:46.114Z DEBUG @wdio/browserstack-service: teardown started 2024-08-20T12:03:46.114Z DEBUG @wdio/browserstack-service: teardown ended 2024-08-20T12:03:46.115Z DEBUG @wdio/utils:shim: Finished to run "after" hook in 2ms

christian-bromann commented 3 weeks ago

@daemontoxa can you verify that you are using v9.0.5 of WebdriverIO?

daemontoxa commented 3 weeks ago

@christian-bromann yep, I've updated to v9.0.5 and after this got the new error with logs which I added to comments 2h ago

Error: Failed to fetch https://hub-cloud.browserstack.com/wd/hub/session: [object Object]

wdio-bot commented 3 weeks ago

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!