search

genotrance / px

An HTTP proxy server to automatically authenticate through an NTLM proxy
MIT License
952 stars 98 forks source link

PX doesn't seem to work with Conda, when CNTLM does #114

Closed plattenschieber closed 2 years ago

plattenschieber commented 4 years ago

I can't get PX to work at our company - even though using the same settings for PX and CNTLM.

Use Case: I try installing the conda package px-proxy via

  1. CNTLM - no issues
  2. PX - It seems that I can't authorize, so here are the logs from conda install -v px-proxy and px 
    
$ conda install -v px-proxy
Collecting package metadata (current_repodata.json): ...working... Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/main/win-64/current_repodata.json

Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/win-64/current_repodata.json Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/noarch/current_repodata.json Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/noarch/current_repodata.json Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/win-64/current_repodata.json Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/noarch/current_repodata.json Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/win-64/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/main/win-64/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/win-64/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/win-64/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/noarch/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/noarch/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/noarch/current_repodata.json Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/win-64/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/main/win-64/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/win-64/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/msys2/noarch/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/noarch/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /conda-forge/win-64/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/noarch/current_repodata.json Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out'))': /pkgs/r/win-64/current_repodata.json

failed Traceback (most recent call last): File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen self._prepare_proxy(conn) File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 932, in _prepare_proxy conn.connect() File "C:\DEV\miniconda\lib\site-packages\urllib3\connection.py", line 317, in connect self._tunnel() File "C:\DEV\miniconda\lib\http\client.py", line 916, in _tunnel (version, code, message) = response._read_status() File "C:\DEV\miniconda\lib\http\client.py", line 267, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "C:\DEV\miniconda\lib\socket.py", line 589, in readinto return self._sock.recv_into(b) socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\DEV\miniconda\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 767, in urlopen response_kw File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 767, in urlopen response_kw File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 767, in urlopen **response_kw File "C:\DEV\miniconda\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "C:\DEV\miniconda\lib\site-packages\urllib3\util\retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/win-64/current_repodata.json (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 485, in fetch_repodata_remote_request timeout=timeout) File "C:\DEV\miniconda\lib\site-packages\requests\sessions.py", line 543, in get return self.request('GET', url, kwargs) File "C:\DEV\miniconda\lib\site-packages\requests\sessions.py", line 530, in request resp = self.send(prep, send_kwargs) File "C:\DEV\miniconda\lib\site-packages\requests\sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "C:\DEV\miniconda\lib\site-packages\requests\adapters.py", line 510, in send raise ProxyError(e, request=request) requests.exceptions.ProxyError: HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/win-64/current_repodata.json (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\DEV\miniconda\lib\site-packages\conda\exceptions.py", line 1079, in call return func(*args, kwargs) File "C:\DEV\miniconda\lib\site-packages\conda\cli\main.py", line 84, in _main exit_code = do_call(args, p) File "C:\DEV\miniconda\lib\site-packages\conda\cli\conda_argparse.py", line 83, in do_call return getattr(module, func_name)(args, parser) File "C:\DEV\miniconda\lib\site-packages\conda\cli\main_install.py", line 20, in execute install(args, parser, 'install') File "C:\DEV\miniconda\lib\site-packages\conda\cli\install.py", line 265, in install should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]), File "C:\DEV\miniconda\lib\site-packages\conda\core\solve.py", line 117, in solve_for_transaction should_retry_solve) File "C:\DEV\miniconda\lib\site-packages\conda\core\solve.py", line 158, in solve_for_diff force_remove, should_retry_solve) File "C:\DEV\miniconda\lib\site-packages\conda\core\solve.py", line 262, in solve_final_state ssc = self._collect_all_metadata(ssc) File "C:\DEV\miniconda\lib\site-packages\conda\common\io.py", line 88, in decorated return f(*args, *kwds) File "C:\DEV\miniconda\lib\site-packages\conda\core\solve.py", line 425, in _collect_all_metadata index, r = self._prepare(prepared_specs) File "C:\DEV\miniconda\lib\site-packages\conda\core\solve.py", line 1021, in _prepare self.subdirs, prepared_specs, self._repodata_fn) File "C:\DEV\miniconda\lib\site-packages\conda\core\index.py", line 228, in get_reduced_index repodata_fn=repodata_fn) File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 105, in query_all result = tuple(concat(executor.map(subdir_query, channel_urls))) File "C:\DEV\miniconda\lib\concurrent\futures_base.py", line 598, in result_iterator yield fs.pop().result() File "C:\DEV\miniconda\lib\concurrent\futures_base.py", line 435, in result return self.get_result() File "C:\DEV\miniconda\lib\concurrent\futures_base.py", line 384, in get_result raise self._exception File "C:\DEV\miniconda\lib\concurrent\futures\thread.py", line 57, in run result = self.fn(self.args, self.kwargs) File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 98, in package_ref_or_match_spec)) File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 110, in query self.load() File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 174, in load _internal_state = self._load() File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 248, in _load repodata_fn=self.repodata_fn) File "C:\DEV\miniconda\lib\site-packages\conda\core\subdir_data.py", line 491, in fetch_repodata_remote_request raise ProxyError() # see #3962 conda.exceptions.ProxyError: Conda cannot proceed due to an error in your proxy configuration. Check for typos and other configuration errors in any '.netrc' file in your home directory, any environment variables ending in '_PROXY', and any other system-wide proxy configuration settings.


and the according px logs:

$ ./px.exe Serving at 127.0.0.1:3127 proc MainProcess MainProcess: MainThread: 1600865583: print_banner: proxy:server = proxy:8081 MainProcess: MainThread: 1600865583: print_banner: proxy:port = 3127 MainProcess: MainThread: 1600865583: print_banner: proxy:listen = 127.0.0.1 MainProcess: MainThread: 1600865583: print_banner: proxy:allow = ... MainProcess: MainThread: 1600865583: print_banner: proxy:gateway = 0 MainProcess: MainThread: 1600865583: print_banner: proxy:hostonly = 0 MainProcess: MainThread: 1600865583: print_banner: proxy:noproxy = MainProcess: MainThread: 1600865583: print_banner: proxy:useragent = MainProcess: MainThread: 1600865583: print_banner: settings:workers = 1 MainProcess: MainThread: 1600865583: print_banner: settings:threads = 1 MainProcess: MainThread: 1600865583: print_banner: settings:idle = 30 MainProcess: MainThread: 1600865583: print_banner: settings:socktimeout = 20.0 MainProcess: MainThread: 1600865583: print_banner: settings:proxyreload = 60 MainProcess: MainThread: 1600865583: print_banner: settings:foreground = 0 MainProcess: MainThread: 1600865583: print_banner: settings:log = 1 MainProcess: MainThread: 1600865583: print_banner: settings:debug = 1 MainProcess: MainThread: 1600865583: print_banner: settings:username = i016214 MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865596: do_CONNECT: Entering MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865596: do_transaction: Entering MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865596: get_destination: repo.anaconda.com:443 MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865596: do_socket_connect: New connection: ('proxy', 8081) MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865596: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865596: do_proxy_type: Searching proxy type MainProcess: Thread_0: 1600865596: do_socket: Entering MainProcess: Thread_0: 1600865596: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865596: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865596: do_socket: Reading response code MainProcess: Thread_0: 1600865596: do_socket: Response code: 407 True MainProcess: Thread_0: 1600865596: do_socket: Reading response headers MainProcess: Thread_0: 1600865596: do_socket: Received Date: Wed, 23 Sep 2020 12:53:16 GMT MainProcess: Thread_0: 1600865596: do_socket: Received Content-Type: text/html MainProcess: Thread_0: 1600865596: do_socket: Received Cache-Control: no-cache MainProcess: Thread_0: 1600865596: do_socket: Received Content-Length: 2473 MainProcess: Thread_0: 1600865596: do_socket: Received X-Frame-Options: deny MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Connection: Keep-Alive MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Authenticate: sanitized (9) MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Authenticate: sanitized (4) MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Authenticate: sanitized (20) MainProcess: Thread_0: 1600865596: do_proxy_type: Auth mechanisms: Negotiate NTLM Basic realm="company" MainProcess: Thread_0: 1600865596: do_proxy_type: Selected: ('proxy', 8081): NTLM MainProcess: Thread_0: 1600865596: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865596: fwd_data: Reading response data MainProcess: Thread_0: 1600865596: fwd_data: Content length 2473 MainProcess: Thread_0: 1600865596: do_socket: Entering MainProcess: Thread_0: 1600865596: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865596: do_socket: Sending extra Proxy-Authorization: sanitized len(77) MainProcess: Thread_0: 1600865596: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865596: do_socket: Reading response code MainProcess: Thread_0: 1600865596: do_socket: Response code: 407 True MainProcess: Thread_0: 1600865596: do_socket: Reading response headers MainProcess: Thread_0: 1600865596: do_socket: Received Date: Wed, 23 Sep 2020 12:53:16 GMT MainProcess: Thread_0: 1600865596: do_socket: Received Content-Type: text/html MainProcess: Thread_0: 1600865596: do_socket: Received Cache-Control: no-cache MainProcess: Thread_0: 1600865596: do_socket: Received Content-Length: 0 MainProcess: Thread_0: 1600865596: do_socket: Received X-Frame-Options: deny MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Connection: Keep-Alive MainProcess: Thread_0: 1600865596: do_socket: Received Proxy-Authenticate: sanitized (141) MainProcess: Thread_0: 1600865596: do_transaction: Auth required MainProcess: Thread_0: 1600865596: do_transaction: Challenged MainProcess: Thread_0: 1600865597: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865597: fwd_data: Reading response data MainProcess: Thread_0: 1600865597: do_socket: Entering MainProcess: Thread_0: 1600865597: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865597: do_socket: Sending extra Proxy-Authorization: sanitized len(429) MainProcess: Thread_0: 1600865597: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865597: do_socket: Reading response code MainProcess: Thread_0: 1600865597: do_socket: Response code: 200 False MainProcess: Thread_0: 1600865597: do_socket: Reading response headers MainProcess: Thread_0: 1600865597: do_CONNECT: Tunneling through proxy MainProcess: Thread_0: 1600865597: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 - MainProcess: MainThread: 1600865597: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865606: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865607: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865618: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865618: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865618: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865618: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865618: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865619: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865619: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865627: do_CONNECT: Proxy connection timeout MainProcess: Thread_0: 1600865627: do_CONNECT: Cleanup proxy connection MainProcess: Thread_0: 1600865627: do_CONNECT: 8646 bytes read, 8646 bytes written MainProcess: Thread_0: 1600865627: do_CONNECT: Done MainProcess: Thread_0: 1600865627: do_CONNECT: Entering MainProcess: Thread_0: 1600865627: do_transaction: Entering MainProcess: Thread_0: 1600865627: get_destination: repo.anaconda.com:443 MainProcess: Thread_0: 1600865627: do_socket_connect: New connection: ('proxy', 8081) MainProcess: Thread_0: 1600865627: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865627: fwd_data: Reading response data MainProcess: Thread_0: 1600865627: do_socket: Entering MainProcess: Thread_0: 1600865627: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865627: do_socket: Sending extra Proxy-Authorization: sanitized len(77) MainProcess: Thread_0: 1600865627: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865627: do_socket: Reading response code MainProcess: Thread_0: 1600865627: do_socket: Response code: 407 True MainProcess: Thread_0: 1600865627: do_socket: Reading response headers MainProcess: Thread_0: 1600865627: do_socket: Received Date: Wed, 23 Sep 2020 12:53:47 GMT MainProcess: Thread_0: 1600865627: do_socket: Received Content-Type: text/html MainProcess: Thread_0: 1600865627: do_socket: Received Cache-Control: no-cache MainProcess: Thread_0: 1600865627: do_socket: Received Content-Length: 0 MainProcess: Thread_0: 1600865627: do_socket: Received X-Frame-Options: deny MainProcess: Thread_0: 1600865627: do_socket: Received Proxy-Connection: Keep-Alive MainProcess: Thread_0: 1600865627: do_socket: Received Proxy-Authenticate: sanitized (141) MainProcess: Thread_0: 1600865627: do_transaction: Auth required MainProcess: Thread_0: 1600865627: do_transaction: Challenged MainProcess: Thread_0: 1600865627: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865627: fwd_data: Reading response data MainProcess: Thread_0: 1600865627: do_socket: Entering MainProcess: Thread_0: 1600865627: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865627: do_socket: Sending extra Proxy-Authorization: sanitized len(429) MainProcess: Thread_0: 1600865627: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865627: do_socket: Reading response code MainProcess: Thread_0: 1600865627: do_socket: Response code: 200 False MainProcess: Thread_0: 1600865627: do_socket: Reading response headers MainProcess: Thread_0: 1600865627: do_CONNECT: Tunneling through proxy MainProcess: Thread_0: 1600865627: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 - MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: MainThread: 1600865633: verify_request: Client address: 127.0.0.1 MainProcess: Thread_0: 1600865642: handle_one_request: Socket error: [WinError 10054] An existing connection was forcibly closed by the remote host MainProcess: Thread_0: 1600865642: handle_one_request: Host disconnected MainProcess: Thread_0: 1600865642: do_CONNECT: Entering MainProcess: Thread_0: 1600865642: do_transaction: Entering MainProcess: Thread_0: 1600865642: get_destination: repo.anaconda.com:443 MainProcess: Thread_0: 1600865642: do_socket_connect: New connection: ('proxy', 8081) MainProcess: Thread_0: 1600865642: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865642: fwd_data: Reading response data MainProcess: Thread_0: 1600865642: do_socket: Entering MainProcess: Thread_0: 1600865642: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865642: do_socket: Sending extra Proxy-Authorization: sanitized len(77) MainProcess: Thread_0: 1600865642: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865642: do_socket: Reading response code MainProcess: Thread_0: 1600865642: do_socket: Response code: 407 True MainProcess: Thread_0: 1600865642: do_socket: Reading response headers MainProcess: Thread_0: 1600865642: do_socket: Received Date: Wed, 23 Sep 2020 12:54:02 GMT MainProcess: Thread_0: 1600865642: do_socket: Received Content-Type: text/html MainProcess: Thread_0: 1600865642: do_socket: Received Cache-Control: no-cache MainProcess: Thread_0: 1600865642: do_socket: Received Content-Length: 0 MainProcess: Thread_0: 1600865642: do_socket: Received X-Frame-Options: deny MainProcess: Thread_0: 1600865642: do_socket: Received Proxy-Connection: Keep-Alive MainProcess: Thread_0: 1600865642: do_socket: Received Proxy-Authenticate: sanitized (141) MainProcess: Thread_0: 1600865643: do_transaction: Auth required MainProcess: Thread_0: 1600865643: do_transaction: Challenged MainProcess: Thread_0: 1600865643: get_response_sspi: pywin32 SSPI MainProcess: Thread_0: 1600865643: fwd_data: Reading response data MainProcess: Thread_0: 1600865643: do_socket: Entering MainProcess: Thread_0: 1600865643: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0 MainProcess: Thread_0: 1600865643: do_socket: Sending extra Proxy-Authorization: sanitized len(429) MainProcess: Thread_0: 1600865643: do_socket: Sending extra b'Proxy-Connection: keep-alive' MainProcess: Thread_0: 1600865643: do_socket: Reading response code MainProcess: Thread_0: 1600865643: do_socket: Response code: 200 False MainProcess: Thread_0: 1600865643: do_socket: Reading response headers MainProcess: Thread_0: 1600865643: do_CONNECT: Tunneling through proxy MainProcess: Thread_0: 1600865643: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 - MainProcess: Thread_0: 1600865643: handle_one_request: Socket error: [WinError 10053] An established connection was aborted by the software in your host machine MainProcess: Thread_0: 1600865643: handle_one_request: Host disconnected ...

This is the config `px.ini`

[proxy] server = proxy:8081 port = 3127 listen = 127.0.0.1 allow = ... gateway = 0 hostonly = 0 noproxy = useragent =

[settings] workers = 1 threads = 1 idle = 30 socktimeout = 20.0 proxyreload = 60 foreground = 0 log = 1 debug = 1 username = username



What could be the issue? Am I missing something?
genotrance commented 4 years ago

Can you try setting auth = Kerberos under the proxy session and retry?

plattenschieber commented 4 years ago

Unfortunately this results in the same failures on px side:

MainProcess: Thread_0: 1600868617: do_CONNECT: Entering
MainProcess: Thread_0: 1600868617: do_transaction: Entering
MainProcess: Thread_0: 1600868617: get_destination: repo.anaconda.com:443
MainProcess: Thread_0: 1600868617: do_socket_connect: New connection: ('proxy', 8081)
MainProcess: Thread_0: 1600868617: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868617: fwd_data: Reading response data
MainProcess: Thread_0: 1600868617: do_socket: Entering
MainProcess: Thread_0: 1600868617: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868617: do_socket: Sending extra Proxy-Authorization: sanitized len(77)
MainProcess: Thread_0: 1600868617: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868617: do_socket: Reading response code
MainProcess: Thread_0: 1600868617: do_socket: Response code: 407 True
MainProcess: Thread_0: 1600868617: do_socket: Reading response headers
MainProcess: Thread_0: 1600868617: do_socket: Received Date: Wed, 23 Sep 2020 13:43:36 GMT
MainProcess: Thread_0: 1600868617: do_socket: Received Content-Type: text/html
MainProcess: Thread_0: 1600868617: do_socket: Received Cache-Control: no-cache
MainProcess: Thread_0: 1600868617: do_socket: Received Content-Length: 0
MainProcess: Thread_0: 1600868617: do_socket: Received X-Frame-Options: deny
MainProcess: Thread_0: 1600868617: do_socket: Received Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1600868617: do_socket: Received Proxy-Authenticate: sanitized (141)
MainProcess: Thread_0: 1600868617: do_transaction: Auth required
MainProcess: Thread_0: 1600868617: do_transaction: Challenged
MainProcess: Thread_0: 1600868617: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868617: fwd_data: Reading response data
MainProcess: Thread_0: 1600868617: do_socket: Entering
MainProcess: Thread_0: 1600868617: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868617: do_socket: Sending extra Proxy-Authorization: sanitized len(429)
MainProcess: Thread_0: 1600868617: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868617: do_socket: Reading response code
MainProcess: Thread_0: 1600868617: do_socket: Response code: 200 False
MainProcess: Thread_0: 1600868617: do_socket: Reading response headers
MainProcess: Thread_0: 1600868617: do_CONNECT: Tunneling through proxy
MainProcess: Thread_0: 1600868617: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 -
MainProcess: Thread_0: 1600868617: handle_one_request: Socket error: [WinError 10053] An established connection was aborted by the software in your host machine
MainProcess: Thread_0: 1600868617: handle_one_request: Host disconnected

Could it be an issue due to windows firewall? But then I don't get, why CNTLM should work.

genotrance commented 4 years ago

Did you restart px after updating the ini file? This isn't the full log.

plattenschieber commented 4 years ago

Sure, I'll just truncated it to the error. Here is the beginning:

$ ./px.exe
Serving at 127.0.0.1:3127 proc MainProcess
MainProcess: MainThread: 1600868546: print_banner: proxy:server = proxy:8081
MainProcess: MainThread: 1600868546: print_banner: proxy:port = 3127
MainProcess: MainThread: 1600868546: print_banner: proxy:listen = 127.0.0.1
MainProcess: MainThread: 1600868546: print_banner: proxy:allow = *.*.*.*
MainProcess: MainThread: 1600868546: print_banner: proxy:gateway = 0
MainProcess: MainThread: 1600868546: print_banner: proxy:hostonly = 0
MainProcess: MainThread: 1600868546: print_banner: proxy:noproxy =
MainProcess: MainThread: 1600868546: print_banner: proxy:useragent =
MainProcess: MainThread: 1600868546: print_banner: proxy:auth = Kerberos
MainProcess: MainThread: 1600868546: print_banner: settings:workers = 1
MainProcess: MainThread: 1600868546: print_banner: settings:threads = 1
MainProcess: MainThread: 1600868546: print_banner: settings:idle = 30
MainProcess: MainThread: 1600868546: print_banner: settings:socktimeout = 20.0
MainProcess: MainThread: 1600868546: print_banner: settings:proxyreload = 60
MainProcess: MainThread: 1600868546: print_banner: settings:foreground = 0
MainProcess: MainThread: 1600868546: print_banner: settings:log = 1
MainProcess: MainThread: 1600868546: print_banner: settings:debug = 1
MainProcess: MainThread: 1600868546: print_banner: settings:username = username
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868565: do_CONNECT: Entering
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868565: do_transaction: Entering
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868565: get_destination: repo.anaconda.com:443
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868565: do_socket_connect: New connection: ('proxy', 8081)
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868565: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868565: do_proxy_type: Searching proxy type
MainProcess: Thread_0: 1600868565: do_socket: Entering
MainProcess: Thread_0: 1600868565: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868565: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868565: do_socket: Reading response code
MainProcess: Thread_0: 1600868566: do_socket: Response code: 407 True
MainProcess: Thread_0: 1600868566: do_socket: Reading response headers
MainProcess: Thread_0: 1600868566: do_socket: Received Date: Wed, 23 Sep 2020 13:42:45 GMT
MainProcess: Thread_0: 1600868566: do_socket: Received Content-Type: text/html
MainProcess: Thread_0: 1600868566: do_socket: Received Cache-Control: no-cache
MainProcess: Thread_0: 1600868566: do_socket: Received Content-Length: 2473
MainProcess: Thread_0: 1600868566: do_socket: Received X-Frame-Options: deny
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Authenticate: sanitized (9)
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Authenticate: sanitized (4)
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Authenticate: sanitized (20)
MainProcess: Thread_0: 1600868566: do_proxy_type: Auth mechanisms: Negotiate NTLM Basic realm="company"
MainProcess: Thread_0: 1600868566: do_proxy_type: Selected: ('proxy', 8081): NTLM
MainProcess: Thread_0: 1600868566: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868566: fwd_data: Reading response data
MainProcess: Thread_0: 1600868566: fwd_data: Content length 2473
MainProcess: Thread_0: 1600868566: do_socket: Entering
MainProcess: Thread_0: 1600868566: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868566: do_socket: Sending extra Proxy-Authorization: sanitized len(77)
MainProcess: Thread_0: 1600868566: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868566: do_socket: Reading response code
MainProcess: Thread_0: 1600868566: do_socket: Response code: 407 True
MainProcess: Thread_0: 1600868566: do_socket: Reading response headers
MainProcess: Thread_0: 1600868566: do_socket: Received Date: Wed, 23 Sep 2020 13:42:45 GMT
MainProcess: Thread_0: 1600868566: do_socket: Received Content-Type: text/html
MainProcess: Thread_0: 1600868566: do_socket: Received Cache-Control: no-cache
MainProcess: Thread_0: 1600868566: do_socket: Received Content-Length: 0
MainProcess: Thread_0: 1600868566: do_socket: Received X-Frame-Options: deny
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1600868566: do_socket: Received Proxy-Authenticate: sanitized (141)
MainProcess: Thread_0: 1600868566: do_transaction: Auth required
MainProcess: Thread_0: 1600868566: do_transaction: Challenged
MainProcess: Thread_0: 1600868566: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868566: fwd_data: Reading response data
MainProcess: Thread_0: 1600868566: do_socket: Entering
MainProcess: Thread_0: 1600868566: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868566: do_socket: Sending extra Proxy-Authorization: sanitized len(429)
MainProcess: Thread_0: 1600868566: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868566: do_socket: Reading response code
MainProcess: Thread_0: 1600868566: do_socket: Response code: 200 False
MainProcess: Thread_0: 1600868566: do_socket: Reading response headers
MainProcess: Thread_0: 1600868566: do_CONNECT: Tunneling through proxy
MainProcess: Thread_0: 1600868566: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 -
MainProcess: MainThread: 1600868576: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868576: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868576: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868576: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868576: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868588: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868588: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868588: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868588: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868588: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868596: do_CONNECT: Proxy connection timeout
MainProcess: Thread_0: 1600868596: do_CONNECT: Cleanup proxy connection
MainProcess: Thread_0: 1600868596: do_CONNECT: 8643 bytes read, 8643 bytes written
MainProcess: Thread_0: 1600868596: do_CONNECT: Done
MainProcess: Thread_0: 1600868596: do_CONNECT: Entering
MainProcess: Thread_0: 1600868596: do_transaction: Entering
MainProcess: Thread_0: 1600868596: get_destination: repo.anaconda.com:443
MainProcess: Thread_0: 1600868596: do_socket_connect: New connection: ('proxy', 8081)
MainProcess: Thread_0: 1600868596: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868596: fwd_data: Reading response data
MainProcess: Thread_0: 1600868596: do_socket: Entering
MainProcess: Thread_0: 1600868596: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868596: do_socket: Sending extra Proxy-Authorization: sanitized len(77)
MainProcess: Thread_0: 1600868596: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868596: do_socket: Reading response code
MainProcess: Thread_0: 1600868596: do_socket: Response code: 407 True
MainProcess: Thread_0: 1600868596: do_socket: Reading response headers
MainProcess: Thread_0: 1600868596: do_socket: Received Date: Wed, 23 Sep 2020 13:43:15 GMT
MainProcess: Thread_0: 1600868596: do_socket: Received Content-Type: text/html
MainProcess: Thread_0: 1600868596: do_socket: Received Cache-Control: no-cache
MainProcess: Thread_0: 1600868596: do_socket: Received Content-Length: 0
MainProcess: Thread_0: 1600868596: do_socket: Received X-Frame-Options: deny
MainProcess: Thread_0: 1600868596: do_socket: Received Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1600868596: do_socket: Received Proxy-Authenticate: sanitized (141)
MainProcess: Thread_0: 1600868596: do_transaction: Auth required
MainProcess: Thread_0: 1600868596: do_transaction: Challenged
MainProcess: Thread_0: 1600868596: get_response_sspi: pywin32 SSPI
MainProcess: Thread_0: 1600868596: fwd_data: Reading response data
MainProcess: Thread_0: 1600868596: do_socket: Entering
MainProcess: Thread_0: 1600868596: do_socket: CONNECT repo.anaconda.com:443 HTTP/1.0
MainProcess: Thread_0: 1600868596: do_socket: Sending extra Proxy-Authorization: sanitized len(429)
MainProcess: Thread_0: 1600868596: do_socket: Sending extra b'Proxy-Connection: keep-alive'
MainProcess: Thread_0: 1600868596: do_socket: Reading response code
MainProcess: Thread_0: 1600868596: do_socket: Response code: 200 False
MainProcess: Thread_0: 1600868596: do_socket: Reading response headers
MainProcess: Thread_0: 1600868596: do_CONNECT: Tunneling through proxy
MainProcess: Thread_0: 1600868596: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 -
MainProcess: MainThread: 1600868602: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868602: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868602: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868602: verify_request: Client address: 127.0.0.1
MainProcess: MainThread: 1600868602: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1600868611: handle_one_request: Socket error: [WinError 10054] An existing connection was forcibly closed by the remote host
...
genotrance commented 4 years ago

Looks like NTLM auth did work:

MainProcess: Thread_0: 1600865597: do_CONNECT: Tunneling through proxy MainProcess: Thread_0: 1600865597: log_message: "CONNECT repo.anaconda.com:443 HTTP/1.0" 200 -

However, conda disconnects immediately after the https tunnel is setup for some reason. Can you try a simple curl call through px to see if that works?

plattenschieber commented 4 years ago

There seems no issue using curl 🤔 My .condarc also uses ssl_verify=false as why I use -k in curl. 

$ curl -v -k https://repo.anaconda.com/pkgs/
* Uses proxy env variable https_proxy == '127.0.0.1:3127'
*   Trying 127.0.0.1:3127...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<html>
<head>
  <title>Anaconda Packages</title>
</head>
<body>
  <h1>Anaconda Packages</h1>
  <p>
    Updates are available as an <a href="./rss.xml">RSS feed</a>.<br />
    Anaconda is brought to you by
    <a href="https://www.anaconda.com">Anaconda, Inc.</a>.<br />
    Anaconda installer archive <a href="../archive/">download page</a>.
    (<a href="../archive/.winzip/">winzip</a>)<br />
    Miniconda installers <a href="../miniconda/">download page</a>.
    (<a href="../miniconda/.winzip/">winzip</a>)<br />
    Please also, check out
    <a href="https://anaconda.org/">anaconda.org</a>.
  </p>

  <p>Access to the Anaconda Packages is subject to our <a href="https://know.anaconda.com/TOS.html">Terms of Service</a>.</p>

  <p>Anaconda Individual Edition is subject to our <a href="https://docs.anaconda.com/anaconda/eula/">Anaconda End User License Agreement</a>.</

  <h2>Anaconda (pkgs/main channel)</h2>

 ... 

  <ul>
    <li>
      Linux x86_64 (64-bit)
      <a href="./archive/linux-64/">archive/linux-* Connected to 127.0.0.1 (127.0.0.1) port 3127 (#0)
...* allocate connect buffer!
* Establish HTTP proxy tunnel to repo.anaconda.com:443
> CONNECT repo.anaconda.com:443 HTTP/1.1
> Host: repo.anaconda.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Server: SimpleHTTP/0.6 Python/3.6.4
< Date: Thu, 24 Sep 2020 14:42:04 GMT
< Proxy-Agent: SimpleHTTP/0.6 Python/3.6.4
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/DEV/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CONNECT phase completed!
* CONNECT phase completed!
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [98 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [6029 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=repo.anaconda.com
*  start date: Sep 24 14:07:37 2020 GMT
*  expire date: Sep 24 14:07:37 2021 GMT
*  issuer: O=xxx; OU=xxx; CN=xxx-Proxy-Issuing-CAxx; C=DE
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x25612a0)
} [5 bytes data]
> GET /pkgs/ HTTP/2
> Host: repo.anaconda.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< age: 1841782
< date: Thu, 24 Sep 2020 14:42:04 GMT
< etag: "xxx"
< cf-ray: xxx-FRA
< server: cloudflare
< expires: Thu, 24 Sep 2020 14:42:34 GMT
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cfduid=xxxxx; expires=Sat, 24-Oct-20 14:42:04 GMT; path=/; domain=.anaconda.com; HttpOnly;ecure
< x-amz-id-2: xxxx
< content-type: text/html
< accept-ranges: bytes
< cache-control: public, max-age=30
< cf-request-id: xxx
< last-modified: Wed, 10 Jun 2020 18:09:06 GMT
< content-length: 11774
< cf-cache-status: HIT
< x-amz-request-id: xxx
< x-amz-version-id: xxx
<
{ [5 bytes data]
100 11774  100 11774    0     0  45459      0 --:--:-- --:--:-- --:--:-- 4563564</a>
        (major support)
    </li>
    <li>
      Linux x86 (32-bit)
      <a href="./archive/linux-32/">archive/linux-32</a>
    </li>
  ...

  <p>
  <b>What about the 'anaconda' channel on anaconda.org at <a href="https://anaconda.org/anaconda">https://anaconda.org/anaconda</a>?</b>
  <br/>
  <br/>
  The Anaconda channel on anaconda.org is a mirror of the packages in pkgs/main. If you are using conda's 'defaults' channel, you do not need thnnel.
  <br/>
  Using the 'anaconda' channel alone, however, will not provide everything in 'defaults' as the 'defaults' channel also includes packages from bplatforms) and pkgs/msys2 (Windows only).
  </p>

</body>
</html>

* Connection #0 to host 127.0.0.1 left intact
plattenschieber commented 4 years ago

Sorry, hit the close button.

ksimonova commented 4 years ago

@plattenschieber did you solve the issue? Could you share how, please? I'm getting same errors when using through conda.

genotrance commented 2 years ago

I just tested px at length with conda but couldn't reproduce this failure. Any testing with the latest versions will be appreciated.