search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.92k stars 258 forks source link

ScrapingBee Proxy Error #313

Closed migoya2020 closed 2 years ago

migoya2020 commented 3 years ago

i am trying to connect a Scrapingbee proxy and i get an error: { "http": "@proxy.scrapingbee.com:8886" "https": "@proxy.scrapingbee.com:8887" "socks5": "@socks.scrapingbee.com:8888" }

The Error:

ValueError: Different settings for http and https proxy servers not supported

migoya2020 commented 3 years ago

Sorry this is what i am using: replace the FAKEAPI

'proxy':{ "url_http": "http://FAKEAPI@proxy.scrapingbee.com:8886", "url_https": "https://FAKEAPI@proxy.scrapingbee.com:8887", "url_socks5": "socks5://FAKEAPI@socks.scrapingbee.com:8888", }

wkeeling commented 3 years ago

You should specify a http proxy or socks proxy rather than both (socks will just override the other). Also, when using a http proxy, the URLs (including the ports) should be the same.

So you should be able to do this:

'proxy': {
    'http': 'http://FAKEAPI@proxy.scrapingbee.com:8887',
    'https': 'https://FAKEAPI@proxy.scrapingbee.com:8887',
}

or this:

'proxy': {
    'http': 'socks5://FAKEAPI@socks.scrapingbee.com:8888',
    'https': 'socks5://FAKEAPI@socks.scrapingbee.com:8888',
}
migoya2020 commented 3 years ago

When i use your solution i get this Error

WARNING:seleniumwire.server:127.0.0.1:40584: request -> HTTP protocol error in client request: Server disconnected

that is for https that i want to use

luxorbis commented 3 years ago

I have the same problem with an auth. proxy (from surfshark). I scrolled through older issues and did not find a solution. Is there still no solve? Btw, I tried 

'verify_ssl': False,
'mitm_http2': False,

with no luck whatsoever. The proxy works with requests package.

Just in case, here is my log:

DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:64590/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64589", "sslProxy": "127.0.0.1:64589", "noProxy": []}, "acceptInsecureCerts": true, "goog:chromeOptions": {"extensions": [], "args": ["--proxy-bypass-list=<-loopback>"]}}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "proxy": {"proxyType": "manual", "httpProxy": "127.0.0.1:64589", "sslProxy": "127.0.0.1:64589", "noProxy": []}, "acceptInsecureCerts": true, "goog:chromeOptions": {"extensions": [], "args": ["--proxy-bypass-list=<-loopback>"]}}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:64590
DEBUG:seleniumwire.server:127.0.0.1:60387: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:60387: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:60387: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:60387: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:60387: clientdisconnect
DEBUG:urllib3.connectionpool:http://127.0.0.1:64590 "POST /session HTTP/1.1" 200 842
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:64590/session/0f1b4bff187917a846f4c05069387b7c/url {"url": "https://whatismyipaddress.com/"}
DEBUG:seleniumwire.server:127.0.0.1:49589: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:61730: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:61730: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:49589: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:49589: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:49589: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:49589: clientdisconnect
WARNING:seleniumwire.server:127.0.0.1:61730: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:61730: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:61730: clientdisconnect
DEBUG:seleniumwire.server:127.0.0.1:64324: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:54135: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:54135: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:64324: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:54135: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:54135: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:54135: clientdisconnect
WARNING:seleniumwire.server:127.0.0.1:64324: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:64324: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:64324: clientdisconnect
DEBUG:seleniumwire.server:127.0.0.1:59199: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:49761: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:59199: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:49761: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:59199: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:59199: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:59199: clientdisconnect
WARNING:seleniumwire.server:127.0.0.1:49761: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:49761: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:49761: clientdisconnect
DEBUG:urllib3.connectionpool:http://127.0.0.1:64590 "POST /session/0f1b4bff187917a846f4c05069387b7c/url HTTP/1.1" 500 1136
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
Traceback (most recent call last):

  File "C:\Users\maxik\Documents\PythonProjects\availabilityBot\vpn_selenium_test.py", line 47, in <module>
    driver.get(url)

  File "C:\Anaconda\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})

  File "C:\Anaconda\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)

  File "C:\Anaconda\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)

WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
  (Session info: chrome=91.0.4472.77)

DEBUG:seleniumwire.server:127.0.0.1:55123: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:55123: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:55123: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:55123: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:55123: clientdisconnect
DEBUG:seleniumwire.server:127.0.0.1:59667: clientconnect
DEBUG:seleniumwire.server:127.0.0.1:59667: serverconnect
  -> ('cy-nic.prod.surfshark.com', 443)
WARNING:seleniumwire.server:127.0.0.1:59667: request
  -> HTTP protocol error in client request: Server disconnected
DEBUG:seleniumwire.server:127.0.0.1:59667: serverdisconnect
  -> ('cy-nic.prod.surfshark.com', 443)
DEBUG:seleniumwire.server:127.0.0.1:59667: clientdisconnect
wkeeling commented 3 years ago

@luxorbis are you able to post your proxy config, including the selenium_wire options your using?

luxorbis commented 3 years ago

Sure

import logging logging.basicConfig(level=logging.DEBUG)

wire_options = { 'verify_ssl': False, 'mitm_http2': False, 'proxy': { 'https':'https://:@cy-nic.prod.surfshark.com', } }

PATH = '' options = webdriver.ChromeOptions() options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36') driver = webdriver.Chrome(executable_path=PATH, options=options, seleniumwire_options=wire_options)

url = "https://whatismyipaddress.com/" driver.get(url)

I did not specify a port since I did not have to in _requests_, but even with some generic ports like 8080 or 443 I get nothing.

The server I am trying to connect to is from Cyprus. I do not know whether it is just a plain proxy server or some special vpn server, but what I do know is that I can connect to it with _requests_, the windows _nslookup_ command gives me this:

Name: cy-nic.prod.surfshark.com Addresses: 195.47.194.99 195.47.194.64

migoya2020 commented 3 years ago

When i use your solution i get this Error

WARNING:seleniumwire.server:127.0.0.1:40584: request -> HTTP protocol error in client request: Server disconnected

that is for https that i want to use

Any idea where this error is coming from?

wkeeling commented 3 years ago

@migoya2020 did you also try adding 'mitm_http2': False to your seleniumwire_options?

wkeeling commented 2 years ago

Closing due to inactivity.

AsadBukharee commented 1 year ago

Any success ?

AsadBukharee commented 1 year ago

**sw_options = { 'proxys': { "http": "http://FAKEAPI:render_js=False&premium_proxy=True@proxy.scrapingbee.com:8886", "https": "https://FAKEAPI:render_js=False&premium_proxy=True@proxy.scrapingbee.com:8887", }, 'mitm_http2': False, }

profile.set_preference("javascript.enabled", False) driver = uc.Chrome(

options=chrome_options, seleniumwire_options=sw_options )**

AsadBukharee commented 1 year ago

this worked for me.

furiiibond commented 1 year ago

This doesn't work for me

furiiibond commented 1 year ago

ValueError: Different settings for http and https proxy servers not supported