search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 256 forks source link

Error code: 502 Message: Bad Gateway. #113

Closed cybermicheal47 closed 2 years ago

cybermicheal47 commented 4 years ago

Error response

Error code: 502

Message: Bad Gateway.

Error code explanation: 502 - Invalid responses from another server/proxy.

this most times happens when i use seleniumwire

wkeeling commented 4 years ago

Does this happen every time you use Selenium Wire, or just some times?

cybermicheal47 commented 4 years ago

Everytime, 98% of the time . I have tried adding connection time out , still same thing

cybermicheal47 commented 4 years ago

But once I use Normal selenium everything works fine but once I use selenium-wire I start getting this issue 98% of the time , most times 100%

wkeeling commented 4 years ago

Are you able to post your code - specifically the part that sets up the webdriver. Also, are you seeing any traceback in your log that you can post?

cybermicheal47 commented 4 years ago

def get_proxies(): global args if args.proxies: proxies=open(args.proxies,'r').read().strip().split('\n') else: proxies=urlopen(Request('https://www.proxy-list.download/api/v1/get?type=http&anon=elite&country=US',headers={'User-Agent':'DeBos/ytviewer'})).read().decode().strip().split('\r\n') log('[INFO] %d proxies successfully loaded!'%len(proxies)) return proxies def bot(id): global args,locks,urls,user_agents,referers,proxies,drivers,watched_videos while True: try: url=choice(urls) with locks[0]: if len(proxies)==0: proxies.extend(get_proxies()) proxy=choice(proxies) proxies.remove(proxy) log('[INFO][%d] Connecting to %s'%(id,proxy)) user_agent=choice(user_agents) if args.user_agent else user_agents(os=('win','android')) log('[INFO][%d] Setting user agent to %s'%(id,user_agent)) if args.slow_start: locks[1].acquire() if system()=='Windows': executable_dir=path_join(environ['APPDATA'],'DeBos','drivers') else: executable_dir=path_join(environ['HOME'],'.DeBos','drivers') seleniumwire_options={ 'verify_ssl': False, 'connection_timeout': None, 'proxy':{ 'http':'http://%s'%proxy, 'https':'https://%s'%proxy, 'no_proxy':'localhost,127.0.0.1' }, 'suppress_connection_errors': False } if args.driver=='chrome': chrome_options=webdriver.ChromeOptions() chrome_options.add_argument('--user-agent={}'.format(user_agent)) chrome_options.add_argument('--mute-audio') chrome_options.add_experimental_option('excludeSwitches',['enable-logging']) if args.headless: chrome_options.add_argument('--headless') if is_root(): chrome_options.add_argument('--no-sandbox') if system()=='Windows': executable_path=path_join(executable_dir,'chromedriver.exe') else: executable_path=path_join(executable_dir, 'chromedriver') driver=webdriver.Chrome(options=chrome_options, seleniumwire_options=seleniumwire_options) #executable_path=executable_path) time.sleep(15)

cybermicheal47 commented 4 years ago

from seleniumwire import webdriver

options = { 'proxy': { 'http': 'http://190.103.178.13:8080', 'https': 'https://190.103.178.13:8080', 'no_proxy': 'localhost,127.0.0.1' } } driver = webdriver.Firefox(seleniumwire_options=options) driver.get("https://whatismyipaddress.com/")

my second example

wkeeling commented 4 years ago

Thanks. I suspect the issue is the connection to the upstream proxy server. I note that you don't specify a username or password so assume that the proxies are unauthenticated?

Have you tried using the browser with the proxies configured directly - to see whether there is no issue with the proxies themselves? So in Chrome, navigate to Preferences > Network Settings and then select "Manual proxy configuration" and enter the IP and port from your second example above. Once entered, try browsing to https://whatismyipaddress.com directly.

cybermicheal47 commented 4 years ago

I have tried it using normal selenium also & it works fine . I have used both paid & free proxies to test this . It has always given me this error

wkeeling commented 4 years ago

OK thanks. It's possible that Selenium Wire is not showing us everything that's going on. Are you able to enable debug logging at the top of your code? If you add the following lines then any tracebacks will come through to the console when you re-run:

import logging
logging.basicConfig(level=logging.DEBUG)
cybermicheal47 commented 4 years ago

INFO:seleniumwire.proxy.client:Created proxy listening on 127.0.0.1:50566 DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50567/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts": true, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:50566", "sslProxy": "127.0.0.1:50566", "noProxy": []}}}, "desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": true, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:50566", "sslProxy": "127.0.0.1:50566", "noProxy": []}, "marionette": true}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:50567 DEBUG:urllib3.connectionpool:http://127.0.0.1:50567 "POST /session HTTP/1.1" 200 833 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50567/session/8ca51396-6fa5-4fde-843f-1cb76ab33182/url {"url": "https://whatismyipaddress.com/"} DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:tracking-protection.cdn.mozilla.net:443 200 DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:content-signature-2.cdn.mozilla.net:443 200 INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/ INFO:seleniumwire.proxy.handler:Capturing request: https://tracking-protection.cdn.mozilla.net/mozstd-trackwhite-digest256/76.0/1589483333 INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:code 502, message Bad Gateway Traceback (most recent call last): File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\seleniumwire\proxy\proxy2.py", line 108, in do_GET conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\seleniumwire\proxy\proxy2.py", line 269, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) socket.timeout: timed out DEBUG:seleniumwire.proxy.handler:http://ocsp.digicert.com/ 502 INFO:seleniumwire.proxy.handler:Capturing request: https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-07-06-00-19-31.chain DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:snippets.cdn.mozilla.net:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://snippets.cdn.mozilla.net/6/Firefox/76.0.1/20200507114007/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/ DEBUG:seleniumwire.proxy.handler:code 502, message Bad Gateway Traceback (most recent call last): File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\seleniumwire\proxy\proxy2.py", line 109, in do_GET res = conn.getresponse() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1322, in getresponse response.begin() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin version, status, reason = self._read_status() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 264, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 669, in readinto return self._sock.recv_into(b) socket.timeout: timed out DEBUG:seleniumwire.proxy.handler:http://ocsp.digicert.com/ 502 DEBUG:seleniumwire.proxy.handler:code 502, message Bad Gateway Traceback (most recent call last): File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\seleniumwire\proxy\proxy2.py", line 108, in do_GET conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) socket.timeout: timed out DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/ 502 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:urllib3.connectionpool:http://127.0.0.1:50567 "POST /session/8ca51396-6fa5-4fde-843f-1cb76ab33182/url HTTP/1.1" 200 14 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request

Process finished with exit code 0

cybermicheal47 commented 4 years ago

i ran the same proxy with selenium and it worked fine

cybermicheal47 commented 4 years ago

this is the debug for SELENIUM everything works fine in it

DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50986/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts": true, "moz:firefoxOptions": {"profile": "UEsDBBQAAAAIAOuhuVADZgKAwAMAAPINAAAHAAAAdXNlci5qc6VWTY/bNhC9F+h/KPbUAjVhO7tA0p42my1QIOgWMRY5EhQ5srimSJYcWqt/36FkJYZtSU5y08d7M5zPxxQhcB+g/PWmCK6hN6ZczRptlWuYSrVnYEVhQN38/guGBL/9+fNP6ZxkBEIIyR6hS2HiGNxCg6JgKRgC3ojCJfyjMMLubiYJXmzhSg8RRcDkWaYQdDkDq1wNB+g1xzmlcben8LUCVkd0tjOjt9YFGLNAn9ST/dzlOa6WTFsM7vnTx5kD5OLU4pXLKpBrHmXQHjklnqOus9s3p5EOjCuhAbwLyIUx/CVyeJXgUTsbL5f/RexFb5i5Hsdi5Zq/7QM9OgOXWRawcWHHpCDOHhbZozALAuy1nCtwBJmCxpbJOPTmGPK0SA0YmbM233ZTVCaU0jlWMWelgUIFCjBwIXMeeaIip4iguISAIzk9YsWYss8vLE0fIMzR+qRwK3JyOezBjrkaCuGDe20Ztj6ncjUJqhB9Dvvdkq3uVmx9+46tl7enkZ9zeC5ybrrV+u0kOEbzrfaJMmFeeM+SV7SgmEjoRprlCHXdglE6eiNaUI+r5eYfh9S5xLgdQ7vGGicUjaKlfgrdmHyuwG5yp2m7nfH2+Pzxng1LZnoVd9A3rG+52b2t7Y5GFyyNOm3wvqfXs2Bax71ETKBdWRJhbDgHVAChcn4AQdJp/9Ih4j1lRI4tjy8rWJTQPVPyrtWEY0otTCPC1XICIsjq0CGzWFPG5HNHDgo3qiMQY96aSEpBqejGvaTNzmUQsZrzU4ExTFYgdx+gFMng+/7PDI+ENDKK3T7ZB+PiXDhH8Ceq/Qia8iJ67cjJrUAYrPp3dt1enzCQfB6dx++lD/7L3Fuf0ndEYNx2mz/KXtW+5STeGS1blj9uUlHrrt4/bODf7uP91xG/aAf26JyJjLaGC4ezT1/n8g2AFlunH92gK9Jau+U5/yNe8t6wXRN36/VDz95IomfVWZ1eMo7whXFyZ3TEmdY4onTd/uBqT+JWaJPvAVZvKzTtPPeq9X6Ot7Tby/Z5fK6OLkGTyR1Eq5eAxWE/LuiugSmO2B44WUKZr3Ss2kX+7elusDBgt5i3xPru7oQ3GCdZo7oY4xpetFz1a+Ly8UhLa48xzzuvnRLm2otYqQ3QE3NB05TwvmGzFs8RIgYtkZ/yLjrLt2lLK7KGuoCw6V7Hkpa7fqeRHZI3SO6hsS9HP3AQDPnA0LI+H134pxJ3Dp5O1Tk+wEund5cP8+o1aQKVjeW4qT1IGojyX9Jh1MdXTlNpcpTH6hLlf1BLAQIUABQAAAAIAOuhuVADZgKAwAMAAPINAAAHAAAAAAAAAAAAAAC2gQAAAAB1c2VyLmpzUEsFBgAAAAABAAEANQAAAOUDAAAAAA=="}}}, "desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": true, "marionette": true, "moz:firefoxOptions": {"profile": "UEsDBBQAAAAIAOuhuVADZgKAwAMAAPINAAAHAAAAdXNlci5qc6VWTY/bNhC9F+h/KPbUAjVhO7tA0p42my1QIOgWMRY5EhQ5srimSJYcWqt/36FkJYZtSU5y08d7M5zPxxQhcB+g/PWmCK6hN6ZczRptlWuYSrVnYEVhQN38/guGBL/9+fNP6ZxkBEIIyR6hS2HiGNxCg6JgKRgC3ojCJfyjMMLubiYJXmzhSg8RRcDkWaYQdDkDq1wNB+g1xzmlcben8LUCVkd0tjOjt9YFGLNAn9ST/dzlOa6WTFsM7vnTx5kD5OLU4pXLKpBrHmXQHjklnqOus9s3p5EOjCuhAbwLyIUx/CVyeJXgUTsbL5f/RexFb5i5Hsdi5Zq/7QM9OgOXWRawcWHHpCDOHhbZozALAuy1nCtwBJmCxpbJOPTmGPK0SA0YmbM233ZTVCaU0jlWMWelgUIFCjBwIXMeeaIip4iguISAIzk9YsWYss8vLE0fIMzR+qRwK3JyOezBjrkaCuGDe20Ztj6ncjUJqhB9Dvvdkq3uVmx9+46tl7enkZ9zeC5ybrrV+u0kOEbzrfaJMmFeeM+SV7SgmEjoRprlCHXdglE6eiNaUI+r5eYfh9S5xLgdQ7vGGicUjaKlfgrdmHyuwG5yp2m7nfH2+Pzxng1LZnoVd9A3rG+52b2t7Y5GFyyNOm3wvqfXs2Bax71ETKBdWRJhbDgHVAChcn4AQdJp/9Ih4j1lRI4tjy8rWJTQPVPyrtWEY0otTCPC1XICIsjq0CGzWFPG5HNHDgo3qiMQY96aSEpBqejGvaTNzmUQsZrzU4ExTFYgdx+gFMng+/7PDI+ENDKK3T7ZB+PiXDhH8Ceq/Qia8iJ67cjJrUAYrPp3dt1enzCQfB6dx++lD/7L3Fuf0ndEYNx2mz/KXtW+5STeGS1blj9uUlHrrt4/bODf7uP91xG/aAf26JyJjLaGC4ezT1/n8g2AFlunH92gK9Jau+U5/yNe8t6wXRN36/VDz95IomfVWZ1eMo7whXFyZ3TEmdY4onTd/uBqT+JWaJPvAVZvKzTtPPeq9X6Ot7Tby/Z5fK6OLkGTyR1Eq5eAxWE/LuiugSmO2B44WUKZr3Ss2kX+7elusDBgt5i3xPru7oQ3GCdZo7oY4xpetFz1a+Ly8UhLa48xzzuvnRLm2otYqQ3QE3NB05TwvmGzFs8RIgYtkZ/yLjrLt2lLK7KGuoCw6V7Hkpa7fqeRHZI3SO6hsS9HP3AQDPnA0LI+H134pxJ3Dp5O1Tk+wEund5cP8+o1aQKVjeW4qT1IGojyX9Jh1MdXTlNpcpTH6hLlf1BLAQIUABQAAAAIAOuhuVADZgKAwAMAAPINAAAHAAAAAAAAAAAAAAC2gQAAAAB1c2VyLmpzUEsFBgAAAAABAAEANQAAAOUDAAAAAA=="}}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:50986 DEBUG:urllib3.connectionpool:http://127.0.0.1:50986 "POST /session HTTP/1.1" 200 729 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50986/session/04583d96-30d8-4cc2-8b80-5f20a8425ef9/url {"url": "https://whatismyipaddress.com/"} DEBUG:urllib3.connectionpool:http://127.0.0.1:50986 "POST /session/04583d96-30d8-4cc2-8b80-5f20a8425ef9/url HTTP/1.1" 200 14 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request

wkeeling commented 4 years ago

Thanks. Seems as though it's definitely connection timeout related. I'll try and reproduce. What version of Selenium Wire are you running?

wkeeling commented 4 years ago

Also, do you have an example open/free proxy? I tried 190.103.178.13:8080 but it seems that requires auth. Don't worry if not, I'll see if I can find one.

cybermicheal47 commented 4 years ago

Version 1.1.0

Here is a site for free proxies list Https://free-proxy-list.net Thank you sir for your effort

cybermicheal47 commented 4 years ago

Just make sure to use only the recent updated proxies in the site

jsvachon2 commented 4 years ago

Hi,

I'm having a very similar issue using Luminati (through Luminati Proxy Manager) as the upstream proxy. Everything works fine using selenium but I can not get it to work through selenium-wire 1.0.12 and Python 3.8.2. My geckodriver is the latest version (0.26) as well.

chintapalli13 commented 4 years ago

Hi,

I started getting the same issue since about a week ago. My driver is very simple and it was working a week ago. Some changes on my side could be new chromedriver/chome I tried with selenium-wire versions 1.0.11 and 1.1.1

driver = seleniumwire.Chrome() driver.header_overrides = { "myheader": "myheadervalue" } I could not even launch any website. With regular selenium everything works fine.

wkeeling commented 4 years ago

@chintapalli13 @jsvachon2 Thanks for your comments.

The "502 Bad Gateway" message is a fairly generic message that means "something went wrong upstream". To determine the root cause, we need to see the tracebacks that come through to the console. To do that you need to enable logging at DEBUG level at the top of your code:

import logging
logging.basicConfig(level=logging.DEBUG)

If you do that and re-run, you should see the tracebacks come through.

cybermicheal47 commented 4 years ago

Hello sir , please any solution yet?

wkeeling commented 4 years ago

I tried a number of the proxies listed on https://free-proxy-list.net - almost all gave a "connection reset", 403 forbidden or just timed out - which manifested as a "502" in Selenium Wire. One proxy did partially work, but returned the page very slowly. I had to set connection_timeout: None in the seleniumwire_options to allow for the slow responses otherwise Selenium Wire itself timed out.

I also tried entering the proxies directly into the browser (so not using Selenium Wire) and I got the same result - either connection reset, timeout or 403 forbidden - when I browsed to any page.

So in summary, the proxy servers seem to be unreliable so it's difficult to see whether there is a problem with Selenium Wire or a problem with the upstream proxy itself. If you're able to find a reliable public proxy that does work in the browser directly, but doesn't work in in Selenium Wire (causes a 502), then let me know and I'll investigate that.

cybermicheal47 commented 4 years ago

Selenium wire is the cause , try those proxies with selenium , you will see that it will work fine. I have tried selenium-wire with paid proxies & it still gave me 502 error

chintapalli13 commented 4 years ago

I ran with logging set to Debug and found the issue is SSL certificate error which manifested 502. I have the openssl installed on my mac, not sure how to resolve this issue. For Selenium I would send an options argument --ignore-certificate-errors to the Chrome. Not sure if selenium-wire options would accept that argument.

Same code with selenium-wire on my windows PC works perfectly

wkeeling commented 4 years ago

@chintapalli13 have you tried setting 'verify_ssl': False in the selenium wire options ?.

chintapalli13 commented 4 years ago

Thank you so much @wkeeling . Setting that option worked perfectly.

wkeeling commented 4 years ago

@cybermicheal47 an update on this; I have managed to reproduce the issue after locating a more stable proxy in the list. It seems that the issue is caused when the underlying HTTP code attempts to establish a SSH connect tunnel. That's happening in Python's http.client module that Selenium Wire uses, so it's fairly low level. I'll keep debugging as it would be good to find a solution. Any updates I'll post here.

cybermicheal47 commented 4 years ago

Alright sir . Thank you

demisx commented 4 years ago

Hi. Any update on this by any chance? My test spider started to fail recently with the same 502 error. It appears to be much slower now crawling just one page compared to what it was before.

# In test-selenium.py
import logging
from scrapy.spiders import Spider
from craigslist.middleware.selenium_request import SeleniumRequest

logging.basicConfig(level=logging.DEBUG)
class TestSeleniumSpider(Spider):
    name = "test_selenium"
    start_urls = ['https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html']
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'craigslist.middleware.SeleniumMiddleware': 543
        },
    }

    def start_requests(self):
        for url in self.start_urls:
            yield SeleniumRequest(
                url=url,
                callback=self.parse)

    def parse(self, response):
        print(response.text)
$ scrapy crawl test_selenium
scrapy crawl test_selenium
INFO:scrapy.utils.log:Scrapy 2.1.0 started (bot: craigslist)
2020-06-10 12:43:01 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: craigslist)
INFO:scrapy.utils.log:Versions: lxml 4.5.1.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.7 (default, Mar 10 2020, 15:43:33) - [Clang 11.0.0 (clang-1100.0.33.17)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Darwin-19.4.0-x86_64-i386-64bit
2020-06-10 12:43:01 [scrapy.utils.log] INFO: Versions: lxml 4.5.1.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.7 (default, Mar 10 2020, 15:43:33) - [Clang 11.0.0 (clang-1100.0.33.17)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Darwin-19.4.0-x86_64-i386-64bit
DEBUG:scrapy.utils.log:Using reactor: twisted.internet.selectreactor.SelectReactor
2020-06-10 12:43:01 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
INFO:scrapy.crawler:Overridden settings:
{'BOT_NAME': 'craigslist',
 'CLOSESPIDER_ERRORCOUNT': 2,
 'CLOSESPIDER_ITEMCOUNT': 3,
 'CLOSESPIDER_PAGECOUNT': 3,
 'CONCURRENT_REQUESTS': 1,
 'COOKIES_ENABLED': False,
 'DEPTH_LIMIT': 2,
 'DOWNLOAD_TIMEOUT': 600,
 'HTTPCACHE_ENABLED': True,
 'NEWSPIDER_MODULE': 'craigslist.spiders',
 'SPIDER_MODULES': ['craigslist.spiders']}
2020-06-10 12:43:01 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'craigslist',
 'CLOSESPIDER_ERRORCOUNT': 2,
 'CLOSESPIDER_ITEMCOUNT': 3,
 'CLOSESPIDER_PAGECOUNT': 3,
 'CONCURRENT_REQUESTS': 1,
 'COOKIES_ENABLED': False,
 'DEPTH_LIMIT': 2,
 'DOWNLOAD_TIMEOUT': 600,
 'HTTPCACHE_ENABLED': True,
 'NEWSPIDER_MODULE': 'craigslist.spiders',
 'SPIDER_MODULES': ['craigslist.spiders']}
INFO:scrapy.extensions.telnet:Telnet Password: d4a0a63a96b0d30d
2020-06-10 12:43:01 [scrapy.extensions.telnet] INFO: Telnet Password: d4a0a63a96b0d30d
INFO:scrapy.middleware:Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2020-06-10 12:43:01 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
DEBUG:root:***> from_crawler()
2020-06-10 12:43:01 [root] DEBUG: ***> from_crawler()
DEBUG:root:***> __init__()
2020-06-10 12:43:01 [root] DEBUG: ***> __init__()
DEBUG:root:***> seleniumwire_options {'verify_ssl': False, 'connection_timeout': None, 'suppress_connection_errors': False}
2020-06-10 12:43:01 [root] DEBUG: ***> seleniumwire_options {'verify_ssl': False, 'connection_timeout': None, 'suppress_connection_errors': False}
INFO:seleniumwire.proxy.client:Created proxy listening on 127.0.0.1:64291
2020-06-10 12:43:01 [seleniumwire.proxy.client] INFO: Created proxy listening on 127.0.0.1:64291
DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:64292/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"prefs": {"profile.managed_default_content_settings.images": 2, "disk-cache-size": 4096}, "extensions": [], "args": ["--blink-settings=imagesEnabled=false", "--disable-notifications", "--incognito", "--disable-extensions", "--disable-gpu", "--disable-infobars", "--disable-web-security", "--no-sandbox"]}, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64291", "sslProxy": "127.0.0.1:64291", "noProxy": ""}, "acceptInsecureCerts": true}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"prefs": {"profile.managed_default_content_settings.images": 2, "disk-cache-size": 4096}, "extensions": [], "args": ["--blink-settings=imagesEnabled=false", "--disable-notifications", "--incognito", "--disable-extensions", "--disable-gpu", "--disable-infobars", "--disable-web-security", "--no-sandbox"]}, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64291", "sslProxy": "127.0.0.1:64291", "noProxy": ""}, "acceptInsecureCerts": true}}
2020-06-10 12:43:02 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:64292/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"prefs": {"profile.managed_default_content_settings.images": 2, "disk-cache-size": 4096}, "extensions": [], "args": ["--blink-settings=imagesEnabled=false", "--disable-notifications", "--incognito", "--disable-extensions", "--disable-gpu", "--disable-infobars", "--disable-web-security", "--no-sandbox"]}, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64291", "sslProxy": "127.0.0.1:64291", "noProxy": ""}, "acceptInsecureCerts": true}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"prefs": {"profile.managed_default_content_settings.images": 2, "disk-cache-size": 4096}, "extensions": [], "args": ["--blink-settings=imagesEnabled=false", "--disable-notifications", "--incognito", "--disable-extensions", "--disable-gpu", "--disable-infobars", "--disable-web-security", "--no-sandbox"]}, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64291", "sslProxy": "127.0.0.1:64291", "noProxy": ""}, "acceptInsecureCerts": true}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:64292
2020-06-10 12:43:02 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:64292
DEBUG:seleniumwire.proxy.handler:accounts.google.com:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: accounts.google.com:443 200
DEBUG:urllib3.connectionpool:http://127.0.0.1:64292 "POST /session HTTP/1.1" 200 817
2020-06-10 12:43:03 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64292 "POST /session HTTP/1.1" 200 817
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
2020-06-10 12:43:03 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
DEBUG:seleniumwire.proxy.handler:http://seleniumwire/scopes 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: http://seleniumwire/scopes 200
INFO:scrapy.middleware:Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'craigslist.middleware.SeleniumMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
2020-06-10 12:43:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'craigslist.middleware.SeleniumMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
INFO:scrapy.middleware:Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-06-10 12:43:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
INFO:scrapy.middleware:Enabled item pipelines:
[]
2020-06-10 12:43:03 [scrapy.middleware] INFO: Enabled item pipelines:
[]
INFO:scrapy.core.engine:Spider opened
2020-06-10 12:43:03 [scrapy.core.engine] INFO: Spider opened
INFO:scrapy.extensions.logstats:Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-06-10 12:43:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
DEBUG:scrapy.extensions.httpcache:Using filesystem cache storage in /Users/dmoore/projects/beocommedia/dimaslist/spiders/craigslist/.scrapy/httpcache
2020-06-10 12:43:03 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in /Users/dmoore/projects/beocommedia/dimaslist/spiders/craigslist/.scrapy/httpcache
INFO:scrapy.extensions.telnet:Telnet console listening on 127.0.0.1:6023
2020-06-10 12:43:03 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
DEBUG:root:***> process_request()
2020-06-10 12:43:03 [root] DEBUG: ***> process_request()
DEBUG:root:***> request.meta: {'download_timeout': 600.0}
2020-06-10 12:43:03 [root] DEBUG: ***> request.meta: {'download_timeout': 600.0}
DEBUG:root:***> request.url: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html
2020-06-10 12:43:03 [root] DEBUG: ***> request.url: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html
DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:64292/session/5955e4dec7ab8b15389d95c3c7daf1ae/url {"url": "https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html"}
2020-06-10 12:43:03 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:64292/session/5955e4dec7ab8b15389d95c3c7daf1ae/url {"url": "https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html"}
DEBUG:seleniumwire.proxy.handler:showlow.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: showlow.craigslist.org:443 200
INFO:seleniumwire.proxy.handler:Capturing request: https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard
INFO:seleniumwire.proxy.handler:Capturing request: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html
INFO:seleniumwire.proxy.handler:Capturing response: https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200 OK
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing response: https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200 OK
DEBUG:seleniumwire.proxy.handler:https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard 200
INFO:seleniumwire.proxy.handler:Capturing response: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html 200 OK
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing response: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html 200 OK
DEBUG:seleniumwire.proxy.handler:https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
DEBUG:seleniumwire.proxy.handler:www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
2020-06-10 12:43:03 [seleniumwire.proxy.handler] DEBUG: www.craigslist.org:443 200
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307
INFO:seleniumwire.proxy.handler:Capturing request: https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21
2020-06-10 12:43:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/styles/cl.css?v=5826de27c327d61d2169c6a45af814f9 200
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/js/postings-concat.min.js?v=ff8523584ac153e83a5e2bb17cdedd21 200
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/styles/leaflet.css?v=d604b055c4c11b9fd29d0c071b394312 200
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/styles/postings.css?v=6060fb8a94f0ca1485b410b185f2a417 200
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/js/leaflet-concat.min.js?v=4e1898d207f2734e9047790b41ed1307 200
INFO:seleniumwire.proxy.handler:Capturing response: https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030 200 OK
2020-06-10 12:43:04 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030 200
2020-06-10 12:43:04 [seleniumwire.proxy.handler] DEBUG: https://www.craigslist.org/js/general-concat.min.js?v=0722ac452635cd755c5ccda7aa169030 200
DEBUG:seleniumwire.proxy.handler:www.gstatic.com:443 200
2020-06-10 12:43:09 [seleniumwire.proxy.handler] DEBUG: www.gstatic.com:443 200
INFO:seleniumwire.proxy.handler:Capturing request: https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin
2020-06-10 12:43:09 [seleniumwire.proxy.handler] INFO: Capturing request: https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin
INFO:seleniumwire.proxy.handler:Capturing response: https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin 200 OK
2020-06-10 12:43:09 [seleniumwire.proxy.handler] INFO: Capturing response: https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin 200 OK
DEBUG:seleniumwire.proxy.handler:https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin 200
2020-06-10 12:43:09 [seleniumwire.proxy.handler] DEBUG: https://www.gstatic.com/chrome/intelligence/assist/ranker/models/translate/2017/03/translate_ranker_model_20170329.pb.bin 200
DEBUG:seleniumwire.proxy.handler:update.googleapis.com:443 200
2020-06-10 12:44:03 [seleniumwire.proxy.handler] DEBUG: update.googleapis.com:443 200
INFO:seleniumwire.proxy.handler:Capturing request: https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a
2020-06-10 12:44:03 [seleniumwire.proxy.handler] INFO: Capturing request: https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a
INFO:seleniumwire.proxy.handler:Capturing response: https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a 200 OK
2020-06-10 12:44:03 [seleniumwire.proxy.handler] INFO: Capturing response: https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a 200 OK
DEBUG:seleniumwire.proxy.handler:https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a 200
2020-06-10 12:44:03 [seleniumwire.proxy.handler] DEBUG: https://update.googleapis.com/service/update2/json?cup2key=10:3494679571&cup2hreq=01937364cd1bde3ccd1dcac70bff4d6dd10882c556b4862287ff1c9aaf78744a 200
DEBUG:urllib3.connectionpool:http://127.0.0.1:64292 "POST /session/5955e4dec7ab8b15389d95c3c7daf1ae/url HTTP/1.1" 500 2293
2020-06-10 12:48:03 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64292 "POST /session/5955e4dec7ab8b15389d95c3c7daf1ae/url HTTP/1.1" 500 2293
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
2020-06-10 12:48:03 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
INFO:scrapy.extensions.logstats:Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-06-10 12:48:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
ERROR:scrapy.core.scraper:Error downloading <GET https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
    response = yield deferred_from_coro(method(request=request, spider=spider))
  File "/Users/dmoore/projects/beocommedia/dimaslist/spiders/craigslist/craigslist/middleware/selenium_middleware.py", line 85, in process_request
    self.driver.get(request.url)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 299.630
  (Session info: chrome=83.0.4103.97)

2020-06-10 12:48:03 [scrapy.core.scraper] ERROR: Error downloading <GET https://showlow.craigslist.org/apa/d/white-mountain-lake-2-bedroom/7139115785.html>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
    response = yield deferred_from_coro(method(request=request, spider=spider))
  File "/Users/dmoore/projects/beocommedia/dimaslist/spiders/craigslist/craigslist/middleware/selenium_middleware.py", line 85, in process_request
    self.driver.get(request.url)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 299.630
  (Session info: chrome=83.0.4103.97)

INFO:scrapy.core.engine:Closing spider (finished)
2020-06-10 12:48:03 [scrapy.core.engine] INFO: Closing spider (finished)
DEBUG:root:***> Closing spider...
2020-06-10 12:48:03 [root] DEBUG: ***> Closing spider...
INFO:seleniumwire.proxy.client:Destroying proxy
2020-06-10 12:48:03 [seleniumwire.proxy.client] INFO: Destroying proxy
DEBUG:seleniumwire.proxy.storage:Cleaning up /Users/dmoore/.seleniumwire/storage-3adf36e1-52a5-4d44-afca-748d8e063972
2020-06-10 12:48:03 [seleniumwire.proxy.storage] DEBUG: Cleaning up /Users/dmoore/.seleniumwire/storage-3adf36e1-52a5-4d44-afca-748d8e063972
DEBUG:selenium.webdriver.remote.remote_connection:DELETE http://127.0.0.1:64292/session/5955e4dec7ab8b15389d95c3c7daf1ae {}
2020-06-10 12:48:03 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:64292/session/5955e4dec7ab8b15389d95c3c7daf1ae {}
DEBUG:urllib3.connectionpool:http://127.0.0.1:64292 "DELETE /session/5955e4dec7ab8b15389d95c3c7daf1ae HTTP/1.1" 200 14
2020-06-10 12:48:03 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64292 "DELETE /session/5955e4dec7ab8b15389d95c3c7daf1ae HTTP/1.1" 200 14
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
2020-06-10 12:48:03 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
INFO:scrapy.statscollectors:Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/selenium.common.exceptions.TimeoutException': 1,
 'elapsed_time_seconds': 300.220856,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 6, 10, 19, 48, 3, 770441),
 'log_count/DEBUG': 40,
 'log_count/ERROR': 1,
 'log_count/INFO': 33,
 'memusage/max': 59342848,
 'memusage/startup': 56930304,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2020, 6, 10, 19, 43, 3, 549585)}
2020-06-10 12:48:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/selenium.common.exceptions.TimeoutException': 1,
 'elapsed_time_seconds': 300.220856,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 6, 10, 19, 48, 3, 770441),
 'log_count/DEBUG': 40,
 'log_count/ERROR': 1,
 'log_count/INFO': 33,
 'memusage/max': 59342848,
 'memusage/startup': 56930304,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2020, 6, 10, 19, 43, 3, 549585)}
INFO:scrapy.core.engine:Spider closed (finished)
2020-06-10 12:48:03 [scrapy.core.engine] INFO: Spider closed (finished)
wkeeling commented 4 years ago

@demisx what version of Selenium Wire are you running?

demisx commented 4 years ago

@wkeeling I'm running 1.2.1 on MacOs X 10.15.4. Here are the versions of other python packages just in case:

$ pip3 list 
Package           Version
----------------- ------------
appdirs           1.4.4
appnope           0.1.0
astroid           2.4.2
attrs             19.3.0
Automat           20.2.0
backcall          0.2.0
black             19.10b0
botocore          1.17.0
CacheControl      0.12.6
certifi           2020.4.5.2
cffi              1.14.0
chardet           3.0.4
chromedriver-py   84.0.4147.30
click             7.1.2
constantly        15.1.0
cryptography      2.9.2
cssselect         1.1.0
decorator         4.4.2
docutils          0.16
future            0.18.2
geocoder          1.38.1
hyperlink         19.0.0
idna              2.9
incremental       17.5.0
ipython           7.15.0
ipython-genutils  0.2.0
isort             4.3.21
itsdangerous      1.1.0
jedi              0.17.0
Jinja2            2.11.2
jmespath          0.10.0
lazy-object-proxy 1.4.3
lxml              4.5.1
MarkupSafe        1.1.1
mccabe            0.6.1
msgpack           1.0.0
packaging         20.4
parsel            1.6.0
parso             0.7.0
pathspec          0.8.0
pexpect           4.8.0
pickleshare       0.7.5
Pillow            7.1.2
pip               20.1.1
pip-api           0.0.14
pip-download      0.2.1
prompt-toolkit    3.0.5
Protego           0.1.16
psycopg2          2.8.5
ptyprocess        0.6.0
pyasn1            0.4.8
pyasn1-modules    0.2.8
pycparser         2.20
PyDispatcher      2.0.5
Pygments          2.6.1
PyHamcrest        2.0.2
pylint            2.5.3
pylru             1.2.0
pyOpenSSL         19.1.0
pyparsing         2.4.7
python-dateutil   2.8.1
pytz              2020.1
queuelib          1.5.0
ratelim           0.1.6
regex             2020.6.8
requests          2.23.0
retrying          1.3.3
rope              0.17.0
scraperapi-sdk    0.2.2
Scrapy            2.1.0
scrapy-crawlera   1.7.0
scrapy-splash     0.7.2
selenium          3.141.0
selenium-wire     1.2.1
service-identity  18.1.0
setuptools        46.0.0
six               1.15.0
SQLAlchemy        1.3.17
toml              0.10.1
traitlets         4.3.3
Twisted           20.3.0
typed-ast         1.4.1
tzlocal           2.1
urllib3           1.25.9
uuid              1.30
w3lib             1.22.0
wcwidth           0.2.4
Werkzeug          1.0.1
wheel             0.34.2
wrapt             1.12.1
zope.interface    5.1.0
wkeeling commented 4 years ago

Thanks @demisx . I notice that although you've set connection_timeout: None in the Selenium Wire options, you're still getting a timeout, although it looks like it may be Selenium itself that's timing out.

You mention that it's slower than it was before and I wonder whether a recent change to keep-alive functionality could be having an effect. Are you able to try switching keep-alive off:

options = {
    'connection_keep_alive': False  
}
driver = webdriver.Chrome(seleniumwire_options=options)
demisx commented 4 years ago

@wkeeling Thank you very much for your prompt responses. What a blessing to see an owner actually responding to your questions. 🙏 Adding 'connection_keep_alive': False indeed made the page crawlable again. Before, it was stuck half way. The Chrome browser would render the page but the spinning circle in the browser tab would still be spinning like it's waiting for some request to complete.

However, still kind of slow and shaky when I have seleniumwire go through a proxy (ScraperApi). I am trying to figure out if it's new Chrome driver related or something else. I've recently updated it to 83.0.4103.39.

wkeeling commented 4 years ago

No worries, glad that's improved things. I'm keen to understand what's going on, since a number of people have reported similar issues with the stability of upstream proxies. From what I can gather so far, it seems to be a problem when the upstream proxy is particularly busy (e.g. a public server). It's possible that Selenium Wire isn't being very efficient with it's socket/connection handling.

Is the proxy you're using publicly available? Be good to see if I can reproduce.

demisx commented 4 years ago

@wkeeling No, I am not using a public proxy. I am using paid service from Scraper API. It used to run smoother, but after a recent python module upgrades, I am often seeing this type of errors (see below). I can give access to the test repo if you communicate me your email address somehow. 

2020-06-12 08:30:45 [seleniumwire.proxy.handler] DEBUG: code 502, message Bad Gateway
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 90, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 375, in connect
    super().connect()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 942, in connect
    self._tunnel()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 916, in _tunnel
    (version, code, message) = response._read_status()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 275, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
wkeeling commented 4 years ago

There's a new version of Selenium Wire v1.2.2 which attempts to better handle upstream errors and should prevent the 502 that you see in the browser. @cybermicheal47 this seems to improve things when using the proxies from the Free Proxies list. You may need to specify a high value for connection_timeout to accommodate slower responses, e.g:

options = {
    'connection_timeout': None  # Don't timeout
}
driver = webdriver.Firefox(seleniumwire_options=options)
cybermicheal47 commented 4 years ago

thank you sir but , it takes too much time to load a page 99% times , the pages doesn't really get loaded

cybermicheal47 commented 4 years ago

C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\python.exe C:/Users/navyo/PycharmProjects/pyqt5learning/selen.py INFO:seleniumwire.proxy.client:Created proxy listening on 127.0.0.1:50266 DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50267/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts": true, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:50266", "sslProxy": "127.0.0.1:50266", "noProxy": []}}}, "desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": true, "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:50266", "sslProxy": "127.0.0.1:50266", "noProxy": []}, "marionette": true}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:50267 DEBUG:urllib3.connectionpool:http://127.0.0.1:50267 "POST /session HTTP/1.1" 200 833 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:50267/session/790ed228-4a3b-4b98-84d5-2c9fb1371191/url {"url": "https://whatismyipaddress.com"} DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ DEBUG:seleniumwire.proxy.handler:firefox.settings.services.mozilla.com:443 200 DEBUG:seleniumwire.proxy.handler:tracking-protection.cdn.mozilla.net:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/ INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt INFO:seleniumwire.proxy.handler:Capturing request: https://tracking-protection.cdn.mozilla.net/allow-flashallow-digest256/1490633678 INFO:seleniumwire.proxy.handler:Capturing response: https://whatismyipaddress.com/ 403 Forbidden DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/ 403 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/styles/cf.errors.css INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/hcaptcha.challenge.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 91, in proxy_request res = conn.getresponse() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1322, in getresponse response.begin() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin version, status, reason = self._read_status() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt INFO:seleniumwire.proxy.handler:Capturing response: http://detectportal.firefox.com/success.txt 200 OK DEBUG:seleniumwire.proxy.handler:http://detectportal.firefox.com/success.txt 200 DEBUG:seleniumwire.proxy.handler:[WinError 10053] An established connection was aborted by the software in your host machine INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 331, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 315, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond INFO:seleniumwire.proxy.handler:Capturing request: http://ocsp.digicert.com/ INFO:seleniumwire.proxy.handler:Capturing response: http://detectportal.firefox.com/success.txt 200 OK DEBUG:seleniumwire.proxy.handler:http://detectportal.firefox.com/success.txt 200 INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt?ipv4 INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt?ipv6 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 331, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 315, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:tracking-protection.cdn.mozilla.net:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://tracking-protection.cdn.mozilla.net/allow-flashallow-digest256/1490633678 INFO:seleniumwire.proxy.handler:Capturing response: https://whatismyipaddress.com/cdn-cgi/scripts/hcaptcha.challenge.js 200 OK DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/cdn-cgi/scripts/hcaptcha.challenge.js 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/styles/cf.errors.css INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 331, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 315, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond INFO:seleniumwire.proxy.handler:Capturing response: http://detectportal.firefox.com/success.txt?ipv4 200 OK DEBUG:seleniumwire.proxy.handler:http://detectportal.firefox.com/success.txt?ipv4 200 INFO:seleniumwire.proxy.handler:Capturing response: https://tracking-protection.cdn.mozilla.net/allow-flashallow-digest256/1490633678 200 OK DEBUG:seleniumwire.proxy.handler:https://tracking-protection.cdn.mozilla.net/allow-flashallow-digest256/1490633678 200 DEBUG:seleniumwire.proxy.handler:[WinError 10053] An established connection was aborted by the software in your host machine ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 91, in proxy_request res = conn.getresponse() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1322, in getresponse response.begin() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin version, status, reason = self._read_status() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response INFO:seleniumwire.proxy.handler:Capturing request: http://detectportal.firefox.com/success.txt?ipv6 INFO:seleniumwire.proxy.handler:Capturing response: https://whatismyipaddress.com/cdn-cgi/styles/cf.errors.css 200 OK DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/cdn-cgi/styles/cf.errors.css 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 331, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 315, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 331, in request super().request(method, url, body, headers=headers) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 315, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond INFO:seleniumwire.proxy.handler:Capturing response: http://detectportal.firefox.com/success.txt?ipv6 200 OK DEBUG:seleniumwire.proxy.handler:http://detectportal.firefox.com/success.txt?ipv6 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js INFO:seleniumwire.proxy.handler:Capturing response: https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js 200 OK DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/cdn-cgi/scripts/cf.common.js 200 ERROR:seleniumwire.proxy.handler:Error making request Traceback (most recent call last): File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 90, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1004, in _send_output self.send(msg) File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 944, in send self.connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\seleniumwire\proxy\proxy2.py", line 367, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1392, in connect super().connect() File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 915, in connect self.sock = self._create_connection( File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 808, in create_connection raise err File "C:\Users\navyo\AppData\Local\Programs\Python\Python38-32\lib\socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond DEBUG:seleniumwire.proxy.handler:whatismyipaddress.com:443 200 INFO:seleniumwire.proxy.handler:Capturing request: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js INFO:seleniumwire.proxy.handler:Capturing response: https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js 200 OK DEBUG:seleniumwire.proxy.handler:https://whatismyipaddress.com/cdn-cgi/scripts/zepto.min.js 200 DEBUG:urllib3.connectionpool:http://127.0.0.1:50267 "POST /session/790ed228-4a3b-4b98-84d5-2c9fb1371191/url HTTP/1.1" 500 264 DEBUG:selenium.webdriver.remote.remote_connection:Finished Request Traceback (most recent call last): File "C:/Users/navyo/PycharmProjects/pyqt5learning/selen.py", line 17, in driver.get("https://whatismyipaddress.com") File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\navyo\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 300000ms

Process finished with exit code 1

cybermicheal47 commented 4 years ago

i tried the same proxy with selenium and it was fast , but with seleniumwire it became 99% slow to the extent it doesn't load a webpage

cybermicheal47 commented 4 years ago

this is the code i used

from seleniumwire import webdriver import logging logging.basicConfig(level=logging.DEBUG)

options = { 'proxy': { 'http': 'http://94.247.16.29:40910', 'https': 'https://94.247.16.29:40910', 'no_proxy': 'localhost,127.0.0.1' }, 'connection_timeout': None

} driver = webdriver.Firefox(seleniumwire_options=options) driver.get("https://whatismyipaddress.com")

cybermicheal47 commented 4 years ago

i added connection timeout to be 15 instead of none and it became faster a bit but is still very slow compare to selenium , it stil takes lots of time to load a webpage

demisx commented 4 years ago

I've tried the latest v1.2.2 without 'connection_keep_alive': False and it still takes forever to crawl one page and eventually times out.

jsvachon2 commented 4 years ago

I just tried with v1.2.2 and it takes forever to crawl one page as well. My options are set as: 'verify_ssl': False, 'connection_timeout': None

wkeeling commented 4 years ago

I need to do some more investigation, but it looks like the issue is low level in Python's http/socket API.

To improve things in the meantime, try switching off keep-alive:

options = {
    'connection_keep_alive': False
    'connection_timeout': 5
}
driver = webdriver.Firefox(seleniumwire_options=options)
jsvachon2 commented 4 years ago
I tried with the keep alive turned off and it did improve a little but it is still really slow Thanks for looking into this Sent from Mail for Windows 10 From: Will KeelingSent: June 18, 2020 3:06 AMTo: wkeeling/selenium-wireCc: jsvachon2; MentionSubject: Re: [wkeeling/selenium-wire] Error code: 502 Message: Bad Gateway. (#113) I need to do some more investigation, but it looks like the issue is low level in Python's http/socket API.To improve things in the meantime, try switching off keep-alive:options = {    'connection_keep_alive': False    'connection_timeout': 5}driver = webdriver.Firefox(seleniumwire_options=options) —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe. 
cybermicheal47 commented 4 years ago

hello please any solution yet

wkeeling commented 4 years ago

@cybermicheal47 I have a solution that I'm currently testing. It will arrive in version 2.0.0 of Selenium Wire, hopefully in the next couple of days at most.

wkeeling commented 4 years ago

Version 2.0.0 of Selenium Wire has now been released which I hope offers a solution to this problem.

The new version introduces the concept of backends which allow you to switch out the component that performs request capture and upstream proxy negotiation.

At present two backends are supported: the default backend that ships with Selenium Wire, and the mitmproxy backend that relies on the open source mitmproxy library.

Using the mitmproxy backend seems to give a general improvement in performance, and also seems to work better/more reliably with the upstream proxies.

So I would suggest switching the backend in your config:

options = {
    'backend': 'mitmproxy',
    'mitmproxy_log_level': 'INFO'
}
driver = webdriver.Firefox(seleniumwire_options=options)  # Also fine to use other browsers e.g. Chrome

To use the backend, you'll need to install mitmproxy:

pip install mitmproxy

I appreciate that this could be considered more of a workaround solution, but hopefully it will at least offer a temporary solution to the problem.

Worth just stressing that this is a new feature so if you discover any issues, let me know and I'll attempt to fix ASAP. Also note that if you're planning on using free proxy servers that these can be very unstable, so please test a number of servers using Selenium Wire and Selenium, before raising an issue.

wkeeling commented 4 years ago

Note that v2.0.0 introduces a couple of API changes:

cybermicheal47 commented 4 years ago

thank you sir

Bool13 commented 4 years ago

I have the same problem. Using a mitm proxy does not help with multithreading, since the same listening port is used. The second backend issues 502.