search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.89k stars 3.05k forks source link

Saucelabs: SSL connection issues to my Saucelabs selenium hub #23896

Open zouhir opened 4 years ago

zouhir commented 4 years ago

Saucelabs provided me with the following URL to create a test:

https://<sauce_user>:<sauce_key>@ondemand.us-west-1.saucelabs.com:443/wd/hub

WPT's sauce runner is unable to connect to that URL, and it throws the following error:

MaxRetryError: HTTPSConnectionPool(host='ondemand.us-west-1.saucelabs.com', port=443): Max retries exceeded with url: /wd/hub/session (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

This issue is not too rare in selenium-web driver world, and usually people attempt to do the following for situations of self-signed certs, I tried but no luck:

1- Changes in the capabilities object: In the following capabilities object, we can ask Webdriver to be OK with self-signed certs: https://github.com/web-platform-tests/wpt/blob/d3ea88514a93b7b69687837de19b1ca6da15cf91/tools/wptrunner/wptrunner/browsers/sauce.py#L58

"acceptInsecureCerts": True

2- changes to Sauceconnect args passing the "--no-ssl-bump-domains=all" flag to the binary executor command in will not result in any change. https://github.com/web-platform-tests/wpt/blob/d3ea88514a93b7b69687837de19b1ca6da15cf91/tools/wptrunner/wptrunner/browsers/sauce.py#L168

Here's a stacktrace dump:

 0:24.28 DEBUG Connecting to Selenium on URL: https://REDACTED:REDACTED@ondemand.us-west-1.saucelabs.com:443/wd/hub
 0:24.28 DEBUG Got command: u'log'
 0:24.66 DEBUG Got command: u'log'
 0:24.66 WARNING Failed to start protocol connection
 0:24.66 DEBUG Got command: u'log'
 0:24.66 WARNING Traceback (most recent call last):
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 47, in setup
    self.connect()
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/tools/wptrunner/wptrunner/executors/executorselenium.py", line 222, in connect
    desired_capabilities=self.capabilities, keep_alive=True)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 374, in execute
    return self._request(command_info[0], url, body=data)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 397, in _request
    resp = self._conn.request(method, url, body=body, headers=headers)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/request.py", line 80, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/request.py", line 171, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/poolmanager.py", line 336, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/zouhir/Projects/scroll-timeline/test/wpt/_venv2/lib/python2.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPSConnectionPool(host='ondemand.us-west-1.saucelabs.com', port=443): Max retries exceeded with url: /wd/hub/session (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
stephenmcgruer commented 4 years ago

@zouhir I want to make sure I understand the flow here, since I'm unfamiliar with Saucelabs. My understanding is that you have a copy of WPT, and a copy of Selenium running on the localhost (e.g. your desktop). Selenium is then controlling a browser that is running inside a VM in the Saucelabs CI system, and it ultimately is pointing that browser at test files that are... where? Is there a separate copy of WPT running inside Saucelabs, or is it loading them from your desktop as well?

The above may not exactly impact on this bug, but I want to make sure I understand the setup when trying to figure out what may be going wrong.

I think the main thing will be to figure out if Saucelabs is serving us a valid SSL cert. I'm no SSL expert (cc @Hexcles who may be able to help), but I used the following command:

echo | openssl s_client -showcerts -servername ondemand.us-west-1.saucelabs.com -connect ondemand.us-west-1.saucelabs.com:443 2>/dev/null | openssl x509 -inform pem -noout -text

I've attached the resulting cert. At first glance, it looks ok? sauce-labs-cert.txt

Also, are you able to test the Selenium relay and see if that works? This looks to be what the code was originally doing, since it used localhost and port 4445 - see https://wiki.saucelabs.com/display/DOCS/Using+the+Selenium+Relay+with+Sauce+Connect+Proxy#UsingtheSeleniumRelaywithSauceConnectProxy-SettingUpYourTeststoUsetheSeleniumRelay . Note that that page specifically mentions problems with Selenium + Python 2.7 + SSL certs.

Hexcles commented 4 years ago

The server cert looks good to me.

Are you running on macOS by the way?

Could you try in Python 2:

r = urllib3.PoolManager().request('GET', 'https://ondemand.us-west-1.saucelabs.com')
print r.data

and see if there's any error?

zouhir commented 4 years ago

@Hexcles using my current device & python 2.7.16, I currently get from the snippet above: {"ready":true,"message":"Sauce Labs automated testing is fully operational"}

Hexcles commented 4 years ago

Yet in the same environment you got the stacktrace above? That's really not expected and I don't have any great clue at the moment.

stephenmcgruer commented 4 years ago

Would it be exactly the same environment? In the wpt case we'd be using our virtualenv I guess?

foolip commented 4 years ago

@zouhir were you able to make any progress on this? What was the command line you tried? We used to run WPT over Sauce Labs for wpt.fyi so this code has definitely worked at one point, but it might have bitrotted since we stopped doing that. (All runs are now performed directly on the VMs with the browsers.)