http.client.RemoteDisconnected: Remote end closed connection without response - When using authenticated proxy and NordVPN

[albrnick]

Hello!

I feel I must be doing something wrong. But I use a basic proxy config of:

  sw_options = {
        'proxy': {
            "https": 'https://username:passwd@us5032.nordvpn.com',
        }
    }

And get the exception:

File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request
    conn.request(self.command, path, req_body, dict(req.headers))
  File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 368, in connect
    super().connect()
  File "/usr/local/lib/python3.7/http/client.py", line 1406, in connect
    super().connect()
  File "/usr/local/lib/python3.7/http/client.py", line 942, in connect
    self._tunnel()
  File "/usr/local/lib/python3.7/http/client.py", line 916, in _tunnel
    (version, code, message) = response._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 275, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

When trying to get the url: https://nordvpn.com/what-is-my-ip/

I've used the same connection string 'username:passwd@us5032.nordvpn.com' in the requests library, and I get the requests page back just fine.

If I change the password to make it fail in requests, I get an error:

requests.exceptions.ProxyError: HTTPSConnectionPool(host='nordvpn.com', port=443): Max retries exceeded with url: /what-is-my-ip/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))

(Just to clarify, that is the requests library error, not seleniun-wire) So I feel really good that the connection information is correct.

I did put some debug code in the seleium-wire to make sure it was parsing the user/pass just fine. And it looks good. And the hashed version that is added to the headers matches what requests does as well...

Any suggestions on how I can further debug this issue?

Thanks for your time! Nick

[wkeeling]

Thanks for raising this. The configuration looks good. I wonder if you could try switching to the mitmproxy backend and see whether that works? That may give some other clues if it also fails to connect.

To switch the backend, first install mitmproxy with:

pip install mitmproxy

Then set the backend option in the settings:

sw_options = {
    'backend': 'mitmproxy'
    'proxy': {
        "https": 'https://username:passwd@us5032.nordvpn.com',
    }
}

[albrnick]

Thanks for the fast response! = ) Crashed again, but with slightly different errors. Looks like 2 errors?

  File "./url_check.py", line 454, in nord
    browser.get( url )
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=https%3A//nordvpn.com/what-is-my-ip/&c=UTF-8&d=We%20can%E2%80%99t%20connect%20to%20the%20server%20at%20nordvpn.com.

url_check.py is my code, and browser is the result of the wire webdriver.Firefox call.

But also got this error from the proxy afterwards:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mitmproxy/proxy/server.py", line 121, in handle
    root_layer()
  File "/usr/local/lib/python3.7/site-packages/mitmproxy/proxy/modes/http_proxy.py", line 23, in __call__
    layer()
TypeError: 'NoneType' object is not callable

mitmproxy has crashed!

Lemme know what I should try next! Thanks! = )

[wkeeling]

Hmmm... from the dns error it looks like the proxy cannot resolve the name of the target server. The second error is a known issue and can be ignored.

Do you see the first error if you try to target other URLs, or is it specific to https://nordvpn.com/what-is-my-up/

[albrnick]

You know, the same url's worked for the requests library with the proxies parameter, and if I remove the 'proxy' entry from the selenium_wire options, it also works.

But debugging the settings we had before, I get pretty much the same error with 2 different urls. www.google.com and www.cnn.com

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=https%3A//www.cnn.com/&c=UTF-8&d=We%20can%E2%80%99t%20connect%20to%20the%20server%20at%20www.cnn.com.

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=https%3A//www.google.com/&c=UTF-8&d=We%20can%E2%80%99t%20connect%20to%20the%20server%20at%20www.google.com.

And just to try and further troubleshoot, I tried on a computer on another network, and still got the same errors.

What can I try next to help figure this out? = ) Thanks for the really fast responses!

[albrnick]

I'm assuming this error means the proxy server is having dns issues? (vs the browser trying to connect to the proxy?)

Is that correct? (Maybe I can dig into the proxy code to get some better errors)

I'm not familiar with how proxies work. Is there some way I can test the proxy alone, w/out using the browser? (ie, start up the proxy server alone, and then pass it requests?)

Thanks for your time!

[wkeeling]

It does suggest that there is some dns issue somewhere, but Selenium Wire's internal proxy merely uses Python's socket API which hooks into the dns lookup machinery provided by the OS. So in theory it would work the same as both the requests library and the browser.

You could perhaps try running mitmproxy stand-alone and configure your browser to talk to that directly. If the error stops then we know the issue must lie with Selenium Wire, but if the error continues then we know that the issue is not related to Selenium Wire and may perhaps exist upstream somewhere or perhaps somewhere in your local environment. It may yield some further clues at least.

To start the proxy you should just be able to run the mitmdump command directly on the command line. I think it listens on port 8080 (it should print the port). Once running configure the proxy settings in your browser to use that port and see how you get on browsing to external sites.

[albrnick]

Thanks for helping out a neophyte :) I did straight browser with mitmdump directly, and looks like it is a mitmproxy issue, I can't get data back, and getting lots of errors like:

<< Cannot establish TLS with client (sni: news.google.com): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert unknown ca')])")

That at least gives me something to chase! Thanks! :)

[albrnick]

Wow! Success! (Well, partly?)

I used the cert talked about her:

https://docs.mitmproxy.org/stable/concepts-certificates/

And converted the .pem file to a .crt file as talked about here:

https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861

And then used the instructions from the comments of that solution, to go to "about:preferences", searched on certificates, and then installed the newly created .crt from mitmproxy! And the test of running the mitmump manually with FF configured with the new cert works! I can browse away! = )

But moving back to the normal selenium wire setup, still having issues. Though I configured my local FF with the new cert, the one that selenium pulls up doesn't have the newly installed cert. (Or it doesn't save the configuration changes?)

Even when I change the browser that Selenium pulls up to include that cert, when I run the program again, that cert isn't there. I guess I'll start seeing what I need to do to add get that cert in there when Selenium pulls up.

[albrnick]

Eugh... So I followed the directions here:

https://stackoverflow.com/a/47015592/357931

And added the cert to the profile. And now when the FF browser comes up from the selenium wire command, I can check, and the cert is indeed still there! :) However, still get the same error:

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=https%3A//nordvpn.com/what-is-my-ip/&c=UTF-8&d=We%20can%E2%80%99t%20connect%20to%20the%20server%20at%20nordvpn.com.

:(

Any other pointers on areas I should look into? Thanks again for all your help!

[wkeeling]

Thanks for trying that suggestion out, and great that you managed to resolve the certificate issue (and that's actually something other people have raised so I will document that I think!).

With the working the standalone mitmdump configuration, are you able to set the upstream proxy to nordvpn - so you'd run the mitmdump command like this:

mitmdump --set mode=upstream:https://us5032.nordvpn.com --set upstream_auth=username:password

(Set your username and password accordingly.)

After doing that see what happens when you try and browse.

[albrnick]

I'll give that a shot! For ease of documentation, here is the command I used to convert the .pem to a .crt that I could then import into FF

openssl x509 -in mitmproxy-ca-cert.pem -inform PEM -out mitmproxy-ca-cert.crt

And here is where I point to the FF profile:

    browser = wire_webdriver.Firefox( seleniumwire_options = sw_options, firefox_profile = '/path/to/data/firefox_profile/' )

[albrnick]

So good news/bad news is that it appears to be in mitmproxy itself. When I use the command you gave, it has the same issue. tSo at this point, it's not something on your side! :) (They set the auth header just fine, I'm digging into why requests works with proxies, but mitmproxy doesn't)

Thanks for all your help!

[wkeeling]

OK good to know - thanks for trying that out. It may also be worth experimenting with other proxies - e.g. Browsermob Proxy as they perhaps might give additional clues. If you do discover what the cause is - do post here! Thanks again.

[albrnick]

Thanks for the tip on Browsermob! That was really helpful. Browsermob was failing as well, so probably not an issue specific to mitmproxy.

Turns out the issue was two fold. requests proxy and mitmproxy assume different ports if none are given. And I was giving none! :) Despite it being listed as a https proxy to requests, it was defaulting to port 80. And turns our Nord does to a proxy on port 80, just an http one though. I believe that mitmproxy / Bowsermob were assuming port 443 if not given.

So that's why they weren't connecting, but requests was! I figured out that Nord's https proxy is on port 88. So once I set that, both mitmproxy and Broswermob worked!

The dns issue we were seeing is due to "DNS over HTTPS" being set in Firefox. And since it couldn't establish an https connection, it also couldn't do a DNS request. And hence the semi-misleading "dns error" message.

Thanks so much for all your help in getting this working! It is much appreciated. :)

[wkeeling]

Great to hear. Now you describe it, it all starts to make sense. Good to know about the DNS over HTTPS issue too. I'd forgotten that Firefox defaults that setting in certain regions.

So is Selenium Wire working for you now when you set the proxy port?

[albrnick]

Yes! Selenium wire is working perfectly! = ) Great library, thanks! :+1:

[wkeeling]

Perfect - glad it's all working.

[UnrealShadowz]

Hi, could you give me a hint. What have you done to get Nord working with mitmproxy?

proxyIP = "de869.nordvpn.com:89" seleniumwireoptions = { 'backend': 'mitmproxy', 'proxy': { "https": 'https://'+login_user +':' +login_pw + '@'+proxyIP, } } driver = webdriver.Chrome(options=chromeoptionsThread,executable_path= dir + "/chromedriver.exe", seleniumwire_options=seleniumwireoptions)

Error: Exception has occurred: WebDriverException Message: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED (Session info: chrome=87.0.4280.88) File "C:\Users\***\AppData\Local\Programs\Python\Python39\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) File "C:\Users\***\AppData\Local\Programs\Python\Python39\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\***\AppData\Local\Programs\Python\Python39\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get self.execute(Command.GET, {'url': url})

[albrnick]

I use port 88 and not port 89. See if that fixes it! :)

[UnrealShadowz]

Thanks for the hint, I have tried it. mitmdump --set mode=upstream:https://de869.nordvpn.com:88 --set upstream_auth=user:pass

It ends with a timeout. If I use, in Chrome, Proxy SwitchyOmega: HTTPS with Port 89 it works. If I test it with mitmdump - Errors: HTTP protocol error in client request: Server disconnected ERR_TUNNEL_CONNECTION_FAILED

[albrnick]

Oh! Make sure the server you chose has the feature proxy_ssl! I pull the list of servers:

https://nordvpn.com/api/server

[albrnick]

And this may help, for mitdump I use -vvvv to get debug output.

And for the upstream_auth I don't use the username / password I use to login, but some weird 16+ character user/pass that is pulled from the account page of nordvpn.

[UnrealShadowz]

I love you! I didn't know that the server need the proxy_ssl feature. Now it works with port 88. I also use the user/pass character strings from the account page. Thanks a lot!

[eqMFqfFd]

This is still an issue. #264