Closed veodko closed 7 months ago
I did some more testing on a local IIS setup and could reproduce the problem. On a SSL enabled site once you enable Windows Authentication
and then set Extended Protection
to Accept
or Required
, curl stops authenticating (meanwhile it works in chrome).
Once you set Extended Protection
to Off
, curl starts working again.
https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/authentication/windowsauthentication/extendedprotection/
This has been already discussed here and apparently a PR has been merged which should fix this, was there some regression or is it simply some low level Windows thing that can't be fixed by curl? https://github.com/curl/curl/issues/3503
cURL shipped with Windows works and gets HTTP 200 response:
curl 8.4.0 (Windows) libcurl/8.4.0 Schannel WinIDN
Release-Date: 2023-10-11
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets
compared to binary downloaded from the internet:
curl 8.5.0 (x86_64-w64-mingw32) libcurl/8.5.0 LibreSSL/3.8.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.58.0 ngtcp2/1.1.0 nghttp3/1.1.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd
I could just use the binary provided by Microsoft, but the problem is that I'm using the cURL lib in PHP (php_curl.dll) and not directly
One difference is that Microsoft's curl build uses Schannel by default, while our official builds use LibreSSL (and used quictls/OpenSSL till 8.4.0_8) by default and offer Schannel as a runtime-selectable option.
How does it work for you with this setting?
set CURL_SSL_BACKEND=schannel
I would have guessed on SSPI, but it looks like both versions have that enabled so the SSPI based NTLM implementation should be used for both.
One difference is that Microsoft's curl build uses Schannel by default, while our official builds use LibreSSL (and used quictls/OpenSSL till 8.4.0_8) by default and offer Schannel as a runtime-selectable option.
How does it work for you with this setting?
set CURL_SSL_BACKEND=schannel
It worked! Indeed this was the issue. Is the SSL backend changeable at runtime (to switch to Schannel when using NTLM automatically) or it must be decided at the beginning?
However I'm still troubled with getting it to work in PHP, since it ships cURL without Schannel support at all, I tried setting the env with putenv()
and windows env vars but it wouldn't work obviously.
Guess I need to re-compile PHP with custom cURL build. I already got to compile cURL with Schannel support, but got some troubles getting it to compile in PHP, no php_curl.dll is generated.
Update: I got it to work, by default PHP is configured to build static libs therefore curl was bundled in the exe, doing configure --with-all-shared --with-curl
gave me php_curl.dll and it worked after just replacing that single DLL. I just wonder, is there any downside to using WinSSL/Schannel instead of LibreSSL in the long run? Can something go wrong when doing normal SSL requests?
In general LibreSSL is more capable than Schannel, and offers these on all Windows versions (e.g. HTTP/3). But depending on your needs Schannel can be just enough.
I'm wondering though why the OpenSSL codepath isn't working for this. It theoretically should, and there may be a bug somewhere.
It worked! Indeed this was the issue. Is the SSL backend changeable at runtime (to switch to Schannel when using NTLM automatically) or it must be decided at the beginning?
It's the latter.
@veodko: Would you be OK to make a test with the last official version (8.4.0_8) using quictls / OpenSSL (as opposed to LibreSSL)? You can find them here: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48689085/artifacts
This would tell whether this is related to the backend change.
Sure, no problem. Since I'm now working on a test env I can even upload the full log with Authorization headers, if it helps in any way.
Unfortunately it didn't work with curl-8.4.0_8-win32-mingw
curl.exe --version
curl 8.4.0 (i686-w64-mingw32) libcurl/8.4.0 OpenSSL/3.1.4 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.58.0 ngtcp2/1.1.0 nghttp3/1.1.0
Release-Date: 2023-10-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd
curl.exe https://192.168.1.206 -v -L -k --negotiate -u test:test
* Trying 192.168.1.206:443...
* Connected to 192.168.1.206 (192.168.1.206) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=DESKTOP-J7QRD3T
* start date: Dec 15 07:43:04 2023 GMT
* expire date: Dec 15 00:00:00 2024 GMT
* issuer: CN=DESKTOP-J7QRD3T
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/2
* Server auth using Negotiate with user 'test'
* [HTTP/2] [1] OPENED stream for https://192.168.1.206/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 192.168.1.206]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==
> User-Agent: curl/8.4.0
> Accept: */*
>
* HTTP/2 stream 1 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
* Downgrades to HTTP/1.1
* Empty reply from server
* Connection #0 to host 192.168.1.206 left intact
* Issue another request to this URL: 'https://192.168.1.206/'
* Found bundle for host: 0xcafb18 [can multiplex]
* Hostname 192.168.1.206 was found in DNS cache
* Trying 192.168.1.206:443...
* Connected to 192.168.1.206 (192.168.1.206) port 443
* ALPN: curl offers http/1.1
* SSL reusing session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=DESKTOP-J7QRD3T
* start date: Dec 15 07:43:04 2023 GMT
* expire date: Dec 15 00:00:00 2024 GMT
* issuer: CN=DESKTOP-J7QRD3T
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/1.1
* Server auth using Negotiate with user 'test'
> GET / HTTP/1.1
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< WWW-Authenticate: Negotiate TlRMTVNTUAACAAAAHgAeADgAAAA1gori8hQMcMUDme0AAAAAAAAAAJgAmABWAAAACgBhSgAAAA9EAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAACAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQAAQAeAEQARQBTAEsAVABPAFAALQBKADcAUQBSAEQAMwBUAAQAHgBEAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAADAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQABwAIALhM0f8qL9oBAAAAAA==
< Date: Fri, 15 Dec 2023 07:47:48 GMT
< Content-Length: 341
<
* Ignoring the response-body
* Connection #1 to host 192.168.1.206 left intact
* Issue another request to this URL: 'https://192.168.1.206/'
* Found bundle for host: 0xcafb18 [serially]
* Re-using existing connection with host 192.168.1.206
* Server auth using Negotiate with user 'test'
> GET / HTTP/1.1
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAH4AAABAAUABlgAAAAAAAABYAAAACAAIAFgAAAAeAB4AYAAAABAAEADWAQAANYKI4goAYUoAAAAPfke7h/FMJ1AtTk2i8zSAxHQAZQBzAHQARABFAFMASwBUAE8AUAAtAFEAUgAxAEkAVAA3AFQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/giSMKqhdt/nJj73KOmR0QEBAAAAAAAAuEzR/yov2gGXQoCpyH+dHwAAAAACAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQAAQAeAEQARQBTAEsAVABPAFAALQBKADcAUQBSAEQAMwBUAAQAHgBEAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAADAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQABwAIALhM0f8qL9oBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAADAAADrxC4xSsujotuXmUTIX3g5atoZpA6OQHqz2yVNF2cEwCgAQAAAAAAAAAAAAAAAAAAAAAAAJACQASABUAFQAUAAvADEAOQAyAC4AMQA2ADgALgAxAC4AMgAwADYAAAAAAAAAAADlXguvkQioMA+4D69iVQOs
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html
< Server: Microsoft-IIS/10.0
< WWW-Authenticate: Negotiate
< WWW-Authenticate: NTLM
< Date: Fri, 15 Dec 2023 07:47:48 GMT
< Content-Length: 1313
<
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1250"/>
<title>401 � Dost�p nieautoryzowany: odmowa dost�pu z powodu nieprawid�owych po�wiadcze�.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
<h3>You do not have permission to view this directory or page using the credentials that you supplied.</h3>
</fieldset></div>
</div>
</body>
</html>
* Connection #1 to host 192.168.1.206 left intact
set CURL_SSL_BACKEND=schannel
curl.exe https://192.168.1.206 -v -L -k --negotiate -u test:test
* Trying 192.168.1.206:443...
* Connected to 192.168.1.206 (192.168.1.206) port 443
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
* using HTTP/2
* Server auth using Negotiate with user 'test'
* [HTTP/2] [1] OPENED stream for https://192.168.1.206/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 192.168.1.206]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==
> User-Agent: curl/8.4.0
> Accept: */*
>
* HTTP/2 stream 1 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
* Downgrades to HTTP/1.1
* Empty reply from server
* Connection #0 to host 192.168.1.206 left intact
* Issue another request to this URL: 'https://192.168.1.206/'
* Found bundle for host: 0x1495c28 [can multiplex]
* Hostname 192.168.1.206 was found in DNS cache
* Trying 192.168.1.206:443...
* Connected to 192.168.1.206 (192.168.1.206) port 443
* schannel: using IP address, SNI is not supported by OS.
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
* Server auth using Negotiate with user 'test'
> GET / HTTP/1.1
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGFKAAAADw==
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< WWW-Authenticate: Negotiate TlRMTVNTUAACAAAAHgAeADgAAAA1goriBVrLBZwIGq8AAAAAAAAAAJgAmABWAAAACgBhSgAAAA9EAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAACAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQAAQAeAEQARQBTAEsAVABPAFAALQBKADcAUQBSAEQAMwBUAAQAHgBEAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAADAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQABwAIAMENmYMrL9oBAAAAAA==
< Date: Fri, 15 Dec 2023 07:51:29 GMT
< Content-Length: 341
<
* Ignoring the response-body
* Connection #1 to host 192.168.1.206 left intact
* Issue another request to this URL: 'https://192.168.1.206/'
* Found bundle for host: 0x1495c28 [serially]
* Re-using existing connection with host 192.168.1.206
* Server auth using Negotiate with user 'test'
> GET / HTTP/1.1
> Host: 192.168.1.206
> Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAH4AAABAAUABlgAAAAAAAABYAAAACAAIAFgAAAAeAB4AYAAAABAAEADWAQAANYKI4goAYUoAAAAPV9g+h/fiDPtA9Y88iJAtu3QAZQBzAHQARABFAFMASwBUAE8AUAAtAFEAUgAxAEkAVAA3AFQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFGR3nAK9IcLcTyvAYeI5egEBAAAAAAAAwQ2Zgysv2gHyctSNo5ev1gAAAAACAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQAAQAeAEQARQBTAEsAVABPAFAALQBKADcAUQBSAEQAMwBUAAQAHgBEAEUAUwBLAFQATwBQAC0ASgA3AFEAUgBEADMAVAADAB4ARABFAFMASwBUAE8AUAAtAEoANwBRAFIARAAzAFQABwAIAMENmYMrL9oBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAADAAADrxC4xSsujotuXmUTIX3g5atoZpA6OQHqz2yVNF2cEwCgAQAJMl/K66S5NhDW6hrxYyKOEJACQASABUAFQAUAAvADEAOQAyAC4AMQA2ADgALgAxAC4AMgAwADYAAAAAAAAAAABL/mB5aK9ybgwRMrgeemMz
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html
< Last-Modified: Fri, 15 Dec 2023 07:40:15 GMT
< Accept-Ranges: bytes
< ETag: "4e2e5ef1292fda1:0"
< Server: Microsoft-IIS/10.0
* Negotiate: noauthpersist -> 0, header part: true
< Persistent-Auth: true
< Date: Fri, 15 Dec 2023 07:51:29 GMT
< Content-Length: 696
<
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>IIS Windows</title>
<style type="text/css">
<!--
body {
color:#000000;
background-color:#0072C6;
margin:0;
}
#container {
margin-left:auto;
margin-right:auto;
text-align:center;
}
a img {
border:none;
}
-->
</style>
</head>
<body>
<div id="container">
<a href="http://go.microsoft.com/fwlink/?linkid=66138&clcid=0x409"><img src="iisstart.png" alt="IIS" width="960" height="600" /></a>
</div>
</body>
</html>* Connection #1 to host 192.168.1.206 left intact
Same on win32 and win64 build. If there is anything more I could check/log then let me know, perhaps some more verbose OpenSSL debug?
Thank you @veodko.
If I understand the log correctly, it fails the same way with OpenSSL, so likely applies to any implementation.
I cannot help with NTLM details, but I can cut test releases, if we have a patch to test.
I can't think of many other ways to fix this than to diagnose exactly where in the code paths the working and the non-working versions go separate ways, where they first make different takes where one is wrong and one is right.
NTLM is old, insecure and annoying to work with. In addition to that, it is mostly used by old legacy proprietary Windows servers, which makes it harder to reproduce and test against. As the only known working curl for this setup is using schannel, it also requires curl to run on Windows to debug this case.
Ideally, this live case could be converted into a test case that can be reproduced elsewhere for everyone. That would make it easier to debug and fix.
Additionally, Microsoft has just deprecated NTLM.
NTLM also doesn't build at all with OpenSSL 3 built with its no-deprecated
option. The necessary crypto is deprecated.
Ref: #12380
Unless we get a recipe to reproduce this issue there is nothing we can do about this. Until then we close this issue.
Well, the recipe to reproduce is already there few messages above and I can't think of any simpler method unfortunately: In IIS, on a SSL enabled site once you enable Windows Authentication and then set Extended Protection to Accept or Required, curl stops authenticating (meanwhile it works in chrome & windows-shipped curl or curl compiled with Schannel). Once you set Extended Protection to Off, curl starts working again. https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/authentication/windowsauthentication/extendedprotection/
This issue doesn't really bother me anymore though, it works fine since I compiled my own Schannel build and used it to compile PHP later. I don't know if it's worth fixing as people mentioned it is being deprecated, although I doubt it will happen soon as a lot of legacy stuff would probably break so NTLM will still be there for a while.
I have the same issue.
I did this
curl.exe https://[client's Exchange On-Premise server]/EWS/Exchange.asmx -v --http1.1 -L -k -c cookie.txt -b cookie.txt -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" --negotiate -u [user_name]
And also many other flags (
--ntlm
,--negotiate --ntlm
,--anyauth
)I expected the following
HTTP 200 response, got HTTP 401 instead.
Opening the link in Chrome (and allegedly, the same request done through Windows shipped version of curl works, I even looked into Chrome's net-internals log and compared the requests, they look the same (also using
Negotiate
in headers, although I couldn't get to see the values and wasn't allowd to install WireShark on client's machine). Doing the request in PHP's curl 7.70 doesn't work either.I tried decoding the NTLM headers with some python tool (in the order of the requests & responses):
It all worked fine until some security related configuration was changed on the IIS server, which they do not want to change back. That's actually not the first time I encountered this problem, had the same situation earlier but IIS config was restored fortunately. Only feedback I got is that NTLMv1 was disabled and v2 must be used, SSLOffloading disabled and some minor SSL changes (minimum bits). I saw there were some issues previously with NTLMv2 in cURL, but all the issues were closed years ago.
There is no proxy, all requests go directly even in Chrome.
curl/libcurl version
curl 8.5.0, NTLM and SSPI enabled
operating system
Windows 10