Unable to connect to older browsers on SauceLabs

jhildenbiddle commented 4 years ago

This is happening with multiple repos, all of which previously worked without issue.

macOS 10.15.5
node v12.15.0
karma v5.1.0
karma-sauce-launcher v4.1.5
mocha v4.1.0 (req'd for IE9 tests)
chai v4.2.0

Connections are made as expected when later versions of Chrome, Edge, Firefox, IE, and Safari are used (although with a deprecation warning as mentioned in #209). Easier versions of every browser fail consistently. For example, here is the console output from testing with Chrome 26 on Windows 10:

16 06 2020 11:53:18.572:INFO [karma-server]: Karma v5.1.0 server started at http://0.0.0.0:9877/
16 06 2020 11:53:18.573:INFO [launcher]: Launching browsers sl_chrome with concurrency unlimited
16 06 2020 11:53:18.583:INFO [launcher]: Starting browser chrome 26.0 (Windows 10) on SauceLabs
(node:28716) DeprecationWarning: Got: "options.rejectUnauthorized" is now deprecated, please use "options.https.rejectUnauthorized"
16 06 2020 11:54:18.589:WARN [launcher]: chrome 26.0 (Windows 10) on SauceLabs have not captured in 60000 ms, killing.
16 06 2020 11:54:18.590:ERROR [SaucelabsLauncher]: Could not quit the Saucelabs selenium connection. Failure message:
16 06 2020 11:54:18.590:ERROR [SaucelabsLauncher]: TypeError: Cannot read property 'deleteSession' of null
    at SaucelabsLauncher.<anonymous> (/Users/john/Dev/canvas-size/node_modules/karma-sauce-launcher/dist/launcher/launcher.js:77:26)
    at Generator.next (<anonymous>)
    at /Users/john/Dev/canvas-size/node_modules/karma-sauce-launcher/dist/launcher/launcher.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/john/Dev/canvas-size/node_modules/karma-sauce-launcher/dist/launcher/launcher.js:4:12)
    at SaucelabsLauncher.<anonymous> (/Users/john/Dev/canvas-size/node_modules/karma-sauce-launcher/dist/launcher/launcher.js:74:31)
    at SaucelabsLauncher.emit (events.js:223:5)
    at SaucelabsLauncher.EventEmitter.emit (domain.js:475:20)
    at SaucelabsLauncher.emitAsync (/Users/john/Dev/canvas-size/node_modules/karma/lib/events.js:52:10)
    at SaucelabsLauncher.BaseLauncher.kill (/Users/john/Dev/canvas-size/node_modules/karma/lib/launchers/base.js:61:27)
2020-06-16T16:56:51.472Z ERROR webdriver: Request failed with status 400 due to Error: Missing or invalid 'desiredCapabilities'
2020-06-16T16:56:51.473Z ERROR webdriver: Error: Missing or invalid 'desiredCapabilities'
    at getErrorFromResponseBody (/Users/john/Dev/canvas-size/node_modules/webdriver/build/utils.js:121:10)
    at WebDriverRequest._request (/Users/john/Dev/canvas-size/node_modules/webdriver/build/request.js:148:56)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)
    at async startWebDriverSession (/Users/john/Dev/canvas-size/node_modules/webdriver/build/utils.js:41:16)
    at async Function.newSession (/Users/john/Dev/canvas-size/node_modules/webdriver/build/index.js:44:23)
    at async Object.remote (/Users/john/Dev/canvas-size/node_modules/webdriverio/build/index.js:66:20)
16 06 2020 11:56:51.473:ERROR [SaucelabsLauncher]: Error: Failed to create session.
Missing or invalid 'desiredCapabilities'
    at startWebDriverSession (/Users/john/Dev/canvas-size/node_modules/webdriver/build/utils.js:45:11)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)

The error descriptions read to me like errors with karma-sauce-launcher:

16 06 2020 11:54:18.590:ERROR [SaucelabsLauncher]: Could not quit the Saucelabs selenium connection. Failure message:
16 06 2020 11:54:18.590:ERROR [SaucelabsLauncher]: TypeError: Cannot read property 'deleteSession' of null

2020-06-16T16:56:51.472Z ERROR webdriver: Request failed with status 400 due to Error: Missing or invalid 'desiredCapabilities'
2020-06-16T16:56:51.473Z ERROR webdriver: Error: Missing or invalid 'desiredCapabilities'

16 06 2020 11:56:51.473:ERROR [SaucelabsLauncher]: Error: Failed to create session.
Missing or invalid 'desiredCapabilities'
    at startWebDriverSession (/Users/john/Dev/canvas-size/node_modules/webdriver/build/utils.js:45:11)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)

I was able to come up with a list of browser/platform combinations that work reliably. Any browser version earlier than the ones listed below will fail to connect.

Chrome 75 / Windows 10
Edge 13 / Windows 10
Firefox 55 / Windows 10
IE10 / Windows 7
Safari / OS X 10.13

One thing I noticed on SauceLabs' Platform Configurator page is an option to select either the "WebDriver (W3C) - Selenium 3/4, Webdriver.io" or "Legacy - Selenium 2/3" API.

Perhaps the issue is related a configuration for the new API being incorrectly being sent to browser/platform combinations that are require the older API?

Thanks!

longlho commented 4 years ago

I got the same issue, last working version was 4.1.4

windwinkel commented 4 years ago

For Version 4.1.5 this change was made: https://github.com/karma-runner/karma-sauce-launcher/commit/7e75e171d695e07af2c4c2e207a5dc579abcfb4e?branch=7e75e171d695e07af2c4c2e207a5dc579abcfb4e&diff=split

It seems to me, that sauce:options is always passed to sauce labs with this change. According to sauce labs however, when sauce:options is passed, it always assumes that the new protocol W3C is used. However older browsers need to use the legacy JSONWP protocol instead.

web-padawan commented 4 years ago

Major dependency bump in a patch release is not that the users expect.

@rcebulko please consider reverting #207 on master and releasing 4.1.6. Then there should be 5.0.0 major release including saucelabs bump.

rcebulko commented 4 years ago

Hi @web-padawan ,

Our project has moved away from Sauce Labs, and I am no longer actively involved on this repo. I briefly joined to try and fix some issues, but was unable to do so given the commits built up over a couple years without release. I believe I still have merge rights, so if you'd like to submit PR rolling by I'll approve it

christian-bromann commented 4 years ago

Major dependency bump in a patch release is not that the users expect.

@web-padawan I don't see major dependency bumps problematic as long as they don't change the behavior of the software. In this case it actually did for people that still test on old browser that don't support the WebDriver protocol. I see two options:

apply the sauce:option capability only if provided capabilities suggest new browser
revert all changes up until v4.1.4, make a new patch release, reapply changes and release as v5

What would you all suggest?

web-padawan commented 4 years ago

Thank you for your feedback!

apply the sauce:option capability only if provided capabilities suggest new browser

Sounds goos to me in case if this would help to fix the errors with older browsers 👍

zyv commented 4 years ago

@web-padawan I don't see major dependency bumps problematic as long as they don't change the behavior of the software. In this case it actually did for people that still test on old browser that don't support the WebDriver protocol. I see two options:

apply the sauce:option capability only if provided capabilities suggest new browser

This sounds like a great solution to me @christian-bromann !

pchorus commented 4 years ago

@christian-bromann Sounds great!

wswebcreation commented 4 years ago

New release that fixes this will come asap

karmarunnerbot commented 4 years ago

:tada: This issue has been resolved in version 4.3.1 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

jhildenbiddle commented 4 years ago

Great to see this fixed. Thank you!

karma-runner / karma-sauce-launcher

Unable to connect to older browsers on SauceLabs #210