search

genotrance / px

An HTTP proxy server to automatically authenticate through an NTLM proxy
MIT License
943 stars 99 forks source link

Version 0.9.0 with Linux doesn't Authenticate using Kerberos #208

Open jl-sitnrw opened 8 months ago

jl-sitnrw commented 8 months ago

Hello,

First thanks for this amazing project, i use it on a daily bases.

Hello,

The Version 0.9.0 doesn't work for me anymore. px doesn't Authenticate with Kerberos anymore. I'm on an Arch Linux with python 3.11.6 for px. PX has it's own venv

❯ klist
Ticket cache: FILE:/tmp/krb5cc_1000_qGNUWe
Default principal: user.name@AD.COMPANY.DE

Valid starting       Expires              Service principal
14.02.2024 10:53:46  14.02.2024 20:53:46  krbtgt/AD.COMPANY.DE@AD.COMPANY.DE
    renew until 15.02.2024 10:53:41
14.02.2024 10:55:50  14.02.2024 20:53:46  HTTP/proxy.ad.company.de@AD.COMPANY.DE
    renew until 15.02.2024 10:53:41
❯ /home/username/.px-proxy/px-proxy/bin/px --config=/home/username/.px.ini --uniqlog --foreground
Python-dotenv could not parse statement starting at line 13
MainProcess: MainThread: 1707913123: /__init__/parse_noproxy/dprint: {'test.ad.company.de'}
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSL: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSPI: False
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SPNEGO: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSAPI: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSNEGOTIATE: False
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_KERBEROS5: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM_WB: False
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: Host: x86_64-pc-linux-gnu
Serving at 127.0.0.1:3128 proc MainProcess
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:server = proxy.ad.company.de:8080
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:port = 3128
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:gateway = 0
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:hostonly = 0
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:allow = *.*.*.*
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:noproxy = 127.0.0.1, 192.168.122.*, 192.168.102.*, 10.10.10.21, test.ad.company.de
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:auth = NEGOTIATE
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:pac = 
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:pac_encoding = utf-8
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:listen = 127.0.0.1
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:useragent = 
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:username = 
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:workers = 4
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:threads = 50
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:idle = 600
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:socktimeout = 20.0
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:proxyreload = 60
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:foreground = 1
MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: settings:log = 3
Python-dotenv could not parse statement starting at line 13
Process-1: MainThread: 1707913123: /set_allow/parse_noproxy/dprint: set()
Python-dotenv could not parse statement starting at line 13
Process-1: MainThread: 1707913123: /__init__/parse_noproxy/dprint: {'test.ad.company.de'}
Process-2: MainThread: 1707913123: /set_allow/parse_noproxy/dprint: set()
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0
Process-2: MainThread: 1707913123: /__init__/parse_noproxy/dprint: {'test.ad.company.de'}
Python-dotenv could not parse statement starting at line 13
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSL: True
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSPI: False
Process-3: MainThread: 1707913123: /set_allow/parse_noproxy/dprint: set()
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSL: True
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SPNEGO: True
Process-3: MainThread: 1707913123: /__init__/parse_noproxy/dprint: {'test.ad.company.de'}
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSPI: False
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSAPI: True
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SPNEGO: True
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSNEGOTIATE: False
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSL: True
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSAPI: True
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_KERBEROS5: True
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSPI: False
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSNEGOTIATE: False
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM: True
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SPNEGO: True
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_KERBEROS5: True
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM_WB: False
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSAPI: True
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM: True
Process-1: MainThread: 1707913123: /__init__/print_curl_version/dprint: Host: x86_64-pc-linux-gnu
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSNEGOTIATE: False
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM_WB: False
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_KERBEROS5: True
Process-2: MainThread: 1707913123: /__init__/print_curl_version/dprint: Host: x86_64-pc-linux-gnu
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM: True
Serving at 127.0.0.1:3128 proc Process-1Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM_WB: False

Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:server = proxy.ad.company.de:8080
Process-3: MainThread: 1707913123: /__init__/print_curl_version/dprint: Host: x86_64-pc-linux-gnu
Serving at 127.0.0.1:3128 proc Process-2
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:port = 3128
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:server = proxy.ad.company.de:8080
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:gateway = 0
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:port = 3128
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:hostonly = 0
Serving at 127.0.0.1:3128 proc Process-3
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:gateway = 0
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:allow = *.*.*.*
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:server = proxy.ad.company.de:8080
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:hostonly = 0
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:noproxy = 127.0.0.1, 192.168.122.*, 192.168.102.*, 10.10.10.21, test.ad.company.de
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:port = 3128
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:allow = *.*.*.*
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:auth = NEGOTIATE
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:gateway = 0
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:noproxy = 127.0.0.1, 192.168.122.*, 192.168.102.*, 10.10.10.21, test.ad.company.de
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac = 
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:hostonly = 0
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:auth = NEGOTIATE
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac_encoding = utf-8
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac = 
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:allow = *.*.*.*
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:listen = 127.0.0.1
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac_encoding = utf-8
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:useragent = 
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:noproxy = 127.0.0.1, 192.168.122.*, 192.168.102.*, 10.10.10.21, test.ad.company.de
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:listen = 127.0.0.1
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:auth = NEGOTIATE
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:username = 
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:useragent = 
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac = 
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:workers = 4
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:username = 
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:pac_encoding = utf-8
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:threads = 50
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:workers = 4
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:idle = 600
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:listen = 127.0.0.1
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:threads = 50
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:socktimeout = 20.0
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:useragent = 
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:idle = 600
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:proxyreload = 60
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: proxy:username = 
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:socktimeout = 20.0
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:foreground = 1
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:workers = 4
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:proxyreload = 60
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:threads = 50
Process-1: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:log = 3
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:foreground = 1
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:idle = 600
Process-2: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:log = 3
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:socktimeout = 20.0
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:proxyreload = 60
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:foreground = 1
Process-3: MainThread: 1707913123: /start_worker/print_banner/dprint: settings:log = 3
MainProcess: MainThread: 1707913133: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
MainProcess: Thread_0: 1707913133: /do_curl/do_client_auth/dprint: No client authentication required
MainProcess: Thread_0: 1707913133: /do_curl/__init__/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: New curl instance
MainProcess: Thread_0: 1707913133: /__init__/_setup/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: HEAD http://www.heise.de/ using HTTP/1.1
MainProcess: Thread_0: 1707913133: /do_HEAD/do_curl/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Path = http://www.heise.de/
MainProcess: Thread_0: 1707913133: /find_proxy_for_url/get_netloc/dprint: netloc = ('www.heise.de', 80), path = /
MainProcess: Thread_0: 1707913133: /do_curl/get_destination/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Proxy = [('proxy.ad.company.de', 8080)]
MainProcess: Thread_0: 1707913133: /do_HEAD/do_curl/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Configuring proxy settings
MainProcess: Thread_0: 1707913133: /do_curl/set_proxy/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Set noproxy to test.ad.company.de
MainProcess: Thread_0: 1707913133: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
MainProcess: Thread_0: 1707913133: /do_curl/bridge/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Setting up bridge
MainProcess: Thread_0: 1707913133: /do_curl/set_headers/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Adding header => Host: www.heise.de
MainProcess: Thread_0: 1707913133: /set_headers/set_useragent/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Setting user agent to curl/8.5.0
MainProcess: Thread_0: 1707913133: /do_curl/set_headers/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Adding header => Accept: */*
MainProcess: Thread_0: 1707913133: /do_curl/set_headers/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Adding header => Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1707913133: /do_curl/set_headers/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Setting headers
MainProcess: Thread_0: 1707913133: /do/add/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Handles = 0
MainProcess: Thread_0: 1707913133: /add/_add_handle/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Add handle
MainProcess: Thread_0: 1707913133: /add/_add_handle/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Added handle
MainProcess: Thread_0: 1707913133: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: Host proxy.ad.company.de:8080 was resolved.
MainProcess: Thread_0: 1707913133: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: IPv6: (none)
MainProcess: Thread_0: 1707913133: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: IPv4: 10.10.10.20
MainProcess: Thread_0: 1707913133: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: Trying 10.10.10.20:8080...
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: Connected to proxy.ad.company.de (10.10.10.20) port 8080
MainProcess: Thread_0: 1707913134: /_debug_callback/save_upstream/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Upstream server = proxy.ad.company.de
MainProcess: Thread_0: 1707913134: /_debug_callback/save_upstream/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Upstream server is proxy
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Sent header => HEAD http://www.heise.de/ HTTP/1.1
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Sent header => Host: www.heise.de
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Sent header => User-Agent: curl/8.5.0
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Sent header => Accept: */*
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Sent header => Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= HTTP/1.1 407 AuthorizedOnly
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Date: Wed, 14 Feb 2024 12:18:54 GMT
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Content-Type: text/html
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Cache-Control: no-cache
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= X-Frame-Options: deny
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Proxy-Connection: Keep-Alive
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Proxy-Authenticate: Negotiate
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Proxy-Authenticate: NTLM
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Received header <= Proxy-Authenticate: Basic sanitized len(26)
MainProcess: Thread_0: 1707913134: /_socket_action/_header_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Done sending headers
MainProcess: Thread_0: 1707913134: /_socket_action/_debug_callback/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Curl info: Connection #0 to host proxy.ad.company.de left intact
MainProcess: Thread_0: 1707913134: /do_curl/do/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Client to authenticate with upstream proxy
MainProcess: Thread_0: 1707913134: /remove/_remove_handle/dprint: 83e40c2eb8b08f1f5dc7d198982e0658414b7d98: Remove handle: 
Process-1: MainThread: 1707913136: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
Process-1: Thread_0: 1707913136: /do_curl/do_client_auth/dprint: No client authentication required
Process-1: Thread_0: 1707913136: /do_curl/__init__/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: New curl instance
Process-1: Thread_0: 1707913136: /__init__/_setup/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: HEAD http://www.heise.de/ using HTTP/1.1
Process-1: Thread_0: 1707913136: /do_HEAD/do_curl/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Path = http://www.heise.de/
Process-1: Thread_0: 1707913136: /find_proxy_for_url/get_netloc/dprint: netloc = ('www.heise.de', 80), path = /
Process-1: Thread_0: 1707913136: /do_curl/get_destination/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Proxy = [('proxy.ad.company.de', 8080)]
Process-1: Thread_0: 1707913136: /do_HEAD/do_curl/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Configuring proxy settings
Process-1: Thread_0: 1707913136: /do_curl/set_proxy/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Set noproxy to test.ad.company.de
Process-1: Thread_0: 1707913136: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
Process-1: Thread_0: 1707913136: /do_curl/bridge/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Setting up bridge
Process-1: Thread_0: 1707913136: /do_curl/set_headers/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Adding header => Host: www.heise.de
Process-1: Thread_0: 1707913136: /set_headers/set_useragent/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Setting user agent to curl/8.5.0
Process-1: Thread_0: 1707913136: /do_curl/set_headers/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Adding header => Accept: */*
Process-1: Thread_0: 1707913136: /do_curl/set_headers/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Adding header => Proxy-Connection: Keep-Alive
Process-1: Thread_0: 1707913136: /do_curl/set_headers/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Setting headers
Process-1: Thread_0: 1707913136: /do/add/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Handles = 0
Process-1: Thread_0: 1707913136: /add/_add_handle/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Add handle
Process-1: Thread_0: 1707913136: /add/_add_handle/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Added handle
Process-1: Thread_0: 1707913136: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: Host proxy.ad.company.de:8080 was resolved.
Process-1: Thread_0: 1707913136: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: IPv6: (none)
Process-1: Thread_0: 1707913136: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: IPv4: 10.10.10.20
Process-1: Thread_0: 1707913136: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: Trying 10.10.10.20:8080...
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: Connected to proxy.ad.company.de (10.10.10.20) port 8080
Process-1: Thread_0: 1707913137: /_debug_callback/save_upstream/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Upstream server = proxy.ad.company.de
Process-1: Thread_0: 1707913137: /_debug_callback/save_upstream/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Upstream server is proxy
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Sent header => HEAD http://www.heise.de/ HTTP/1.1
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Sent header => Host: www.heise.de
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Sent header => User-Agent: curl/8.5.0
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Sent header => Accept: */*
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Sent header => Proxy-Connection: Keep-Alive
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= HTTP/1.1 407 AuthorizedOnly
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Date: Wed, 14 Feb 2024 12:18:57 GMT
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Content-Type: text/html
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Cache-Control: no-cache
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= X-Frame-Options: deny
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Proxy-Connection: Keep-Alive
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Proxy-Authenticate: Negotiate
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Proxy-Authenticate: NTLM
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Received header <= Proxy-Authenticate: Basic sanitized len(26)
Process-1: Thread_0: 1707913137: /_socket_action/_header_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Done sending headers
Process-1: Thread_0: 1707913137: /_socket_action/_debug_callback/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Curl info: Connection #0 to host proxy.ad.company.de left intact
Process-1: Thread_0: 1707913137: /do_curl/do/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Client to authenticate with upstream proxy
Process-1: Thread_0: 1707913137: /remove/_remove_handle/dprint: 0e58126f72ddd9c7b6c04549b08e63595d4a3010: Remove handle: 
Process-3: MainThread: 1707913146: /_handle_request_noblock/verify_request/dprint: Client address: 127.0.0.1
Process-3: Thread_0: 1707913146: /do_curl/do_client_auth/dprint: No client authentication required
Process-3: Thread_0: 1707913146: /do_curl/__init__/dprint: 61650d20c265816068974f9536bc14d2ac891746: New curl instance
Process-3: Thread_0: 1707913146: /__init__/_setup/dprint: 61650d20c265816068974f9536bc14d2ac891746: CONNECT ts01-lanner-lion.cloudsink.net:443 using HTTP/1.1
Process-3: Thread_0: 1707913146: /_setup/set_tunnel/dprint: 61650d20c265816068974f9536bc14d2ac891746: HTTP proxy tunneling = True
Process-3: Thread_0: 1707913146: /do_CONNECT/do_curl/dprint: 61650d20c265816068974f9536bc14d2ac891746: Path = ts01-lanner-lion.cloudsink.net:443
Process-3: Thread_0: 1707913146: /find_proxy_for_url/get_netloc/dprint: netloc = ('ts01-lanner-lion.cloudsink.net', 443), path = /
Process-3: Thread_0: 1707913146: /do_curl/get_destination/dprint: 61650d20c265816068974f9536bc14d2ac891746: Proxy = [('proxy.ad.company.de', 8080)]
Process-3: Thread_0: 1707913146: /do_CONNECT/do_curl/dprint: 61650d20c265816068974f9536bc14d2ac891746: Configuring proxy settings
Process-3: Thread_0: 1707913146: /do_curl/set_proxy/dprint: 61650d20c265816068974f9536bc14d2ac891746: Set noproxy to test.ad.company.de
Process-3: Thread_0: 1707913146: /set_proxy/set_tunnel/dprint: 61650d20c265816068974f9536bc14d2ac891746: HTTP proxy tunneling = False
Process-3: Thread_0: 1707913146: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth
Process-3: Thread_0: 1707913146: /do_curl/set_headers/dprint: 61650d20c265816068974f9536bc14d2ac891746: Adding header => Host: ts01-lanner-lion.cloudsink.net:443
Process-3: Thread_0: 1707913146: /set_headers/set_useragent/dprint: 61650d20c265816068974f9536bc14d2ac891746: Setting user agent to CrowdStrike Falcon Sensor
Process-3: Thread_0: 1707913146: /do_curl/set_headers/dprint: 61650d20c265816068974f9536bc14d2ac891746: Delaying headers
Process-3: Thread_0: 1707913146: /do/add/dprint: 61650d20c265816068974f9536bc14d2ac891746: Handles = 0
Process-3: Thread_0: 1707913146: /add/_add_handle/dprint: 61650d20c265816068974f9536bc14d2ac891746: Add handle
Process-3: Thread_0: 1707913146: /add/_add_handle/dprint: 61650d20c265816068974f9536bc14d2ac891746: Added handle
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: Host proxy.ad.company.de:8080 was resolved.
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: IPv6: (none)
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: IPv4: 10.10.10.20
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: Trying 10.10.10.20:8080...
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: Connected to proxy.ad.company.de (10.10.10.20) port 8080
Process-3: Thread_0: 1707913146: /_debug_callback/save_upstream/dprint: 61650d20c265816068974f9536bc14d2ac891746: Upstream server = proxy.ad.company.de
Process-3: Thread_0: 1707913146: /_debug_callback/save_upstream/dprint: 61650d20c265816068974f9536bc14d2ac891746: Upstream server is proxy
Process-3: Thread_0: 1707913146: /_socket_action/_debug_callback/dprint: 61650d20c265816068974f9536bc14d2ac891746: Curl info: Connection #0 to host proxy.ad.company.de left intact
Process-3: Thread_0: 1707913146: /do_curl/do/dprint: 61650d20c265816068974f9536bc14d2ac891746: Getting active socket
Process-3: Thread_0: 1707913146: /do_curl/select/dprint: 61650d20c265816068974f9536bc14d2ac891746: Starting select loop
Process-3: Thread_0: 1707913146: /do_curl/select/dprint: 61650d20c265816068974f9536bc14d2ac891746: Sending original client headers

The curl for the request with http_proxy set to localhost:3128

❯ curl --head http://proxy.sit.nrw
HTTP/1.1 407 AuthorizedOnly
Date: Wed, 14 Feb 2024 10:25:55 GMT
Content-Type: text/html
Cache-Control: no-cache
X-Frame-Options: deny
Proxy-Connection: Keep-Alive
Proxy-Authenticate: Negotiate
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="proxy.ad.company.de"

The curl with 0.8.4 same config:

❯ curl --head http://www.heise.de
HTTP/1.1 301 Moved Permanently
Age: 20
Date: Wed, 14 Feb 2024 11:59:45 GMT
Vary: X-Export-Format, X-Export-Agent, X-Export-IAP
Server: nginx
Location: https://www.heise.de/
Content-Type: text/html
Cache-Control: no-store
Last-Modified: Wed, 14 Feb 2024 11:59:45 GMT
Content-Length: 162
x-frame-options: DENY
Proxy-Connection: Keep-Alive
X-Hacc-Refreshed: 
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

Curl itself works with this setting if the company Proxy is set via http_proxy

❯  curl --head --proxy-negotiate -u :  http://www.heise.de
HTTP/1.1 301 Moved Permanently
Age: 2
Date: Wed, 14 Feb 2024 12:53:22 GMT
Vary: X-Export-Format, X-Export-Agent, X-Export-IAP
Server: nginx
Location: https://www.heise.de/
Content-Type: text/html
Cache-Control: no-store
Last-Modified: Wed, 14 Feb 2024 12:53:22 GMT
Content-Length: 162
x-frame-options: DENY
Proxy-Connection: Keep-Alive
X-Hacc-Refreshed: 
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

I don't know where to look.

Any Ideas

genotrance commented 8 months ago

Two things stand out - first:

Python-dotenv could not parse statement starting at line 13

Looks like there's a .env file in your folder with something in it. Not sure if that's distracting the configuration. Second:

MainProcess: MainThread: 1707913123: /run_pool/print_banner/dprint: proxy:username = 
...
MainProcess: Thread_0: 1707913133: /do_curl/set_curl_auth/dprint: SSPI not available and no username configured - no auth

Looks like you have no username configured for px to use for kerberos auth. Can you share what your config looks like without any personal info?

jl-sitnrw commented 8 months ago

I saw that too, couldn't find anything wrong. I never did set the username as I use Kerberos only. NTLM isn't allowed.

As far as I know ,SSPI is windows only. On linux GSSAPI should be used. 

❯ curl -V
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

My config:

❯ grep -Ev "^;" /home/username/.px.ini

[proxy]
server = proxy.ad.company.de:8080
port = 3128
gateway = 0
hostonly = 0
allow = *.*.*.*
noproxy = 127.0.0.1, 192.168.122.*, 192.168.102.*, 10.10.10.21, test.ad.company.de
auth = NEGOTIATE

[settings]
workers = 4
threads = 50
idle = 600
socktimeout = 20.0
proxyreload = 60
foreground = 1
log = 4
jl-sitnrw commented 8 months ago

I found it in handler.py. I'll send a MR, it's an easy fix.

genotrance commented 7 months ago

Thanks for the PR though we need to make it a bit smarter to detect if GSSAPI is available via curl.

MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SSPI: False
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_SPNEGO: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSAPI: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_GSSNEGOTIATE: False
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_KERBEROS5: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM: True
MainProcess: MainThread: 1707913123: /__init__/print_curl_version/dprint: CURL_VERSION_NTLM_WB: False

I test px on several setups and GSSAPI is not available on many of them. Same applies on Windows too - it should check for SSPI before setting username to :. I need to see which values from the above values are needed for auth to work - also wonder if there's a way to detect if NTLM/Kerberos is actually setup with a username/password or just generally available. On Windows, I presume SSPI is always available since you log in but not as informed on Kerberos on Linux.

Meanwhile, I am also working on migrating px to use mcurl which was extracted out of this project and made standalone. I made sure those binaries include GSSAPI so we can be assured that it will always be available once we move. Just need additional changes to check for availability.

AndreasALoew commented 6 months ago

Just by the way, similar issues on Windows: @genotrance , are you aware of https://github.com/curl/curl/issues/13056?

currently, it needs a manual update of curl for Windows in px-0.9.2 in order to get SSPI, Kerberos and SPNEGO working again on Windows platform - so please definitely update curl for Windows with the next px release...

thx a million! 😃

AndreasALoew commented 6 months ago

oops, sorry - turns out I completely missed the 0.9.2 release so far... 😞

so everything should be fine again for Windows as wrt this comment...

Luffy610 commented 6 months ago

Can someone please help me to find some documentation on how this HTTP server principal is created for keytab I guess I am still missing that bit

genotrance commented 6 months ago

oops, sorry - turns out I completely missed the 0.9.2 release so far... 😞

so everything should be fine again for Windows as wrt this comment...

Yes - this was fixed in https://github.com/genotrance/px/issues/212.

jl-sitnrw commented 6 months ago

I test px on several setups and GSSAPI is not available on many of them. Same applies on Windows too - it should check for SSPI before setting username to :. I need to see which values from the above values are needed for auth to work - also wonder if there's a way to detect if NTLM/Kerberos is actually setup with a username/password or just generally available. On Windows, I presume SSPI is always available since you log in but not as informed on Kerberos on Linux.

I have to object partially here. SSPI Is available, but you do not have a Kerberos Ticket/Login if you're not in a Windows Domain. Curl still will have "CURL_VERSION_SSPI" set to "True", because it is build with support for it. According to Documentation:

       --negotiate
              (HTTP) Enables Negotiate (SPNEGO) authentication.

              This option requires a library built with GSS-API or SSPI support. Use -V, --version to see if your curl supports GSS-API/SSPI or SPNEGO.

Either of GSS-API or SSPI is fine.

To detect if there is a TGT and an Available serviceprincipal for "HTTP/proxyname@DOMAIN" without an additional lib would be hard if possible at all. I didn't deep dive, but I think curl just tries to get the Token for the service from os and fails to auth, if it doesn't get it.

genotrance commented 1 month ago

I'll use the libcurl docs to cover all relevant scenarios. Looks like we need to check GSS-API, NTLM, SPNEGO and SSPI. Checking for username is not required since libcurl will automatically use current credentials. If it fails, we will need to detect the failure from libcurl and log that.

If user does not provide a username, we will attempt single sign-on if it is possible via libcurl. Else user will have to provide both username and password.