search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

Hide all the [Errno 61] Connection refused and [Errno 65] No route to host? #192

Closed 097115 closed 3 years ago

097115 commented 3 years ago

Is it possible NOT to print out every OSError: [Errno 65] No route to host and ConnectionRefusedError: [Errno 61] Connection refused error?

Tried setting suppress_connection_errors to True like this:

sw_options = {
    'suppress_connection_errors': True
}
driver = webdriver.Firefox(capabilities=firefox_capabilities,
        options=firefox_options,
        seleniumwire_options=sw_options,
        service_log_path='/dev/null')

...but somehow this didn't help.

Python 3.9.1 selenium 3.141.0 selenium-wire 3.0.4

wkeeling commented 3 years ago

Thanks for this. I'm sure we can improve the error logging here. Just so I'm looking at the correct place in the code, is there a traceback you can share, or alternatively a URL that exhibits the problem so I can reproduce?

097115 commented 3 years ago

Hi! Thanks for your answer!

For some reason, I can't reliably reproduce but something like this:

#!/usr/bin/env python3

from seleniumwire import webdriver

firefox_options = webdriver.FirefoxOptions()
firefox_options.headless = True
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
firefox_capabilities['pageLoadStrategy'] = 'eager'
sw_options = {
    'suppress_connection_errors': True
}
driver = webdriver.Firefox(capabilities=firefox_capabilities,
        options=firefox_options,
        seleniumwire_options=sw_options,
        service_log_path='/dev/null')

u = 'https://www.google.com'
x = '/html/body/div[3]'
driver.get(u)

driver.implicitly_wait(30)
try:
    driver.find_element_by_xpath(x).click()
except:
    print("ERROR: the layer wasn't found!")

driver.quit()

...may effect in something like this:

Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host
Error making request
Traceback (most recent call last):
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 85, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/097115/Library/Python/3.9/lib/python/site-packages/seleniumwire/proxy/proxy2.py", line 364, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 65] No route to host

At first I thought it's my hosts file blocking some crap connections. But now I'm not really sure...

wkeeling commented 3 years ago

Thanks - that's useful to have. I actually tried reproducing by modifying my local hosts file to send certain hosts to an invalid IP, but it didn't seem to generate the OSErrors. Anyway, now I have the tracebacks above, I'll look at catching and suppressing the error with the suppress_connection_errors option. Many thanks.

wkeeling commented 3 years ago

Version 3.0.5 now available which suppresses all connection related errors by default. These can be made visible by setting the suppress_connection_errors option to False in the seleniumwire_options.

097115 commented 3 years ago

Thanks, Will!