search

jeroen / curl

A Modern and Flexible Web Client for R
https://jeroen.r-universe.dev/curl
Other
219 stars 70 forks source link

"schannel: next InitializeSecurityContext failed" error after update to 3.0 #127

Closed maalsol closed 1 week ago

maalsol commented 7 years ago

After updating curl to the latest version, I started getting the following error:

Error in curl::curl_fetch_memory(url, handle = handle) : schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid.

Before updating, the following code worked (using httr):

httr::set_config(config(ssl_verifypeer = 0L))
response <- GET(requestURL, authenticate(":", ":", "ntlm"))

The certificate of the intranet site I'm connecting to is self-signed and it has been imported to the Windows certificate store.

I will appreciate any help on this.

jeroen commented 7 years ago

Perhaps related to https://github.com/curl/curl/issues/568

jeroen commented 7 years ago

Can you try with httr::set_config(config(ssl_verifypeer = 0L, ssl_verifyhost = 0L))

maalsol commented 7 years ago

Hi, jeroen, sorry for the late reply. I'm on windows 7.

The output using verbose() and httr::set_config(config(ssl_verifypeer = 0L, ssl_verifyhost = 0L) is: (replaced site name with <site> and the IP address with <IP>)

*  Hostname in DNS cache was stale, zapped
 *  Trying <IP>...
 *  TCP_NODELAY set
 *  Connected to <site> (<IP>) port 443 (#11)
 *  schannel: SSL/TLS connection with  <site> port 443 (step 1/3)
 *  schannel: disabled server certificate revocation checks
 *  schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
 *  schannel: sending initial handshake data: sending 164 bytes...
 *  schannel: sent initial handshake data: sent 164 bytes
 *  schannel: SSL/TLS connection with <site> port 443 (step 2/3)
 *  schannel: failed to receive handshake, need more data
 *  schannel: SSL/TLS connection with <site> port 443 (step 2/3)
 *  schannel: encrypted data got 2798
 *  schannel: encrypted data buffer: offset 2798 length 4096
 *  schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid
 *  Closing connection 11
 *  schannel: shutting down SSL/TLS connection with <site> port 443
 *  schannel: clear security context handle
 Error in curl::curl_fetch_memory(url, handle = handle) : 
   schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid
jeroen commented 7 years ago

This is a bit of a guess but I suspect that the server runs TLS 1.2 and Windows 7 has not enabled this by default. Can you try this: https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in (note the easy fix link in there)

maalsol commented 7 years ago

Thanks for the suggestion, but TLS 1.2, was already enabled. I ran the "easy fix" tool anyway (just in case) and verified the registry keys, but still, no luck.

jeroen commented 7 years ago

Does this work for you (TLS 1.2 only): curl::curl_fetch_memory("https://rud.is/")

maalsol commented 7 years ago

Yes, I think it does: 

curl::curl_fetch_memory("https://rud.is/")
$url
[1] "https://rud.is/"

$status_code
[1] 200

$headers
  [1] 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 6e 67 69 6e 78 2f 31 2e 31 31 2e 32 0d 0a 44
 [41] 61 74 65 3a 20 57 65 64 2c 20 31 38 20 4f 63 74 20 32 30 31 37 20 32 30 3a 33 31 3a 31 39 20 47 4d 54 0d 0a 43 6f 6e 74
 [81] 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 74 2f 68 74 6d 6c 0d 0a 4c 61 73 74 2d 4d 6f 64 69 66 69 65 64 3a 20 54 68 75 2c
[121] 20 33 31 20 41 75 67 20 32 30 31 37 20 31 36 3a 33 38 3a 35 30 20 47 4d 54 0d 0a 56 61 72 79 3a 20 41 63 63 65 70 74 2d
[161] 45 6e 63 6f 64 69 6e 67 0d 0a 45 54 61 67 3a 20 57 2f 22 35 39 61 38 33 62 39 61 2d 31 31 62 34 22 0d 0a 45 78 70 69 72
[201] 65 73 3a 20 46 72 69 2c 20 30 31 20 53 65 70 20 32 30 31 37 20 31 36 3a 33 38 3a 35 30 20 47 4d 54 0d 0a 43 61 63 68 65
[241] 2d 43 6f 6e 74 72 6f 6c 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 53 74 72 69 63 74 2d 54 72 61 6e 73 70 6f 72 74 2d 53 65 63
[281] 75 72 69 74 79 3a 20 6d 61 78 2d 61 67 65 3d 33 31 35 33 36 30 30 30 3b 20 69 6e 63 6c 75 64 65 53 75 62 44 6f 6d 61 69
[321] 6e 73 3b 20 70 72 65 6c 6f 61 64 0d 0a 43 6f 6e 74 65 6e 74 2d 53 65 63 75 72 69 74 79 2d 50 6f 6c 69 63 79 3a 20 66 72
[361] 61 6d 65 2d 61 6e 63 65 73 74 6f 72 73 20 27 73 65 6c 66 27 0d 0a 58 2d 46 72 61 6d 65 2d 4f 70 74 69 6f 6e 73 3a 20 53
[401] 41 4d 45 4f 52 49 47 49 4e 0d 0a 58 2d 50 6f 77 65 72 65 64 2d 42 79 3a 20 3c 33 0d 0a 43 6f 6e 74 65 6e 74 2d 53 65 63
[441] 75 72 69 74 79 2d 50 6f 6c 69 63 79 3a 20 64 65 66 61 75 6c 74 2d 73 72 63 20 2a 20 27 73 65 6c 66 27 20 64 61 74 61 3a
[481] 20 27 75 6e 73 61 66 65 2d 69 6e 6c 69 6e 65 27 20 27 75 6e 73 61 66 65 2d 65 76 61 6c 27 3b 0d 0a 58 2d 43 6f 6e 74 65
[521] 6e 74 2d 53 65 63 75 72 69 74 79 2d 50 6f 6c 69 63 79 3a 20 64 65 66 61 75 6c 74 2d 73 72 63 20 2a 20 27 73 65 6c 66 27
[561] 20 64 61 74 61 3a 20 27 75 6e 73 61 66 65 2d 69 6e 6c 69 6e 65 27 20 27 75 6e 73 61 66 65 2d 65 76 61 6c 27 3b 0d 0a 58
[601] 2d 57 65 62 4b 69 74 2d 43 53 50 3a 20 64 65 66 61 75 6c 74 2d 73 72 63 20 2a 20 27 73 65 6c 66 27 20 64 61 74 61 3a 20
[641] 27 75 6e 73 61 66 65 2d 69 6e 6c 69 6e 65 27 20 27 75 6e 73 61 66 65 2d 65 76 61 6c 27 3b 0d 0a 58 2d 58 53 53 2d 50 72
[681] 6f 74 65 63 74 69 6f 6e 3a 20 31 3b 20 6d 6f 64 65 3d 62 6c 6f 63 6b 0d 0a 58 2d 43 6f 6e 74 65 6e 74 2d 54 79 70 65 2d
[721] 4f 70 74 69 6f 6e 73 3a 20 6e 6f 73 6e 69 66 66 0d 0a 43 6f 6e 74 65 6e 74 2d 45 6e 63 6f 64 69 6e 67 3a 20 67 7a 69 70
[761] 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 31 32 31 0d 0a 41 67 65 3a 20 32 0d 0a 56 69 61 3a 20 48 54 54
[801] 50 53 2f 31 2e 31 20 57 53 47 50 72 6f 78 79 0d 0a 0d 0a

$modified
[1] "2017-08-31 12:38:50 EDT"

$times
     redirect    namelookup       connect   pretransfer starttransfer         total 
        0.000         0.172         0.188         0.359         0.437         0.468 

$content
   [1] 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 20 50 55 42 4c 49 43 20 22 2d 2f 2f 57 33 43 2f 2f 44 54 44 20 58 48 54 4d 4c
  [41] 20 31 2e 30 20 54 72 61 6e 73 69 74 69 6f 6e 61 6c 2f 2f 45 4e 22 20 22 68 74 74 70 3a 2f 2f 77 77 77 2e 77 33 2e 6f 72
  [81] 67 2f 54 52 2f 78 68 74 6d 6c 31 2f 44 54 44 2f 78 68 74 6d 6c 31 2d 74 72 61 6e 73 69 74 69 6f 6e 61 6c 2e 64 74 64 22
 [121] 3e 0a 3c 68 74 6d 6c 20 78 6d 6c 6e 73 3d 22 68 74 74 70 3a 2f 2f 77 77 77 2e 77 33 2e 6f 72 67 2f 31 39 39 39 2f 78 68
 [161] 74 6d 6c 22 3e 0a 3c 68 65 61 64 3e 0a 3c 6d 65 74 61 20 63 68 61 72 73 65 74 3d 22 55 54 46 2d 38 22 2f 3e 0a 3c 6d 65
 [201] 74 61 20 6e 61 6d 65 3d 22 76 69 65 77 70 6f 72 74 22 20 63 6f 6e 74 65 6e 74 3d 22 77 69 64 74 68 3d 64 65 76 69 63 65
 [241] 2d 77 69 64 74 68 2c 20 69 6e 69 74 69 61 6c 2d 73 63 61 6c 65 3d 31 22 20 2f 3e 0a 3c 6d 65 74 61 20 6e 61 6d 65 3d 22
 [281] 74 68 65 6d 65 2d 63 6f 6c 6f 72 22 20 63 6f 6e 74 65 6e 74 3d 22 23 34 35 37 35 62 34 22 2f 3e 0a 3c 21 2d 2d 20 6d 65
 [321] 74 61 20 6e 61 6d 65 3d 22 76 69 65 77 70 6f 72 74 22 20 63 6f 6e 74 65 6e 74 3d 22 75 73 65 72 2d 73 63 61 6c 61 62 6c
 [361] 65 3d 66 61 6c 73 65 22 20 2f 20 2d 2d 3e 0a 3c 6d 65 74 61 20 6e 61 6d 65 3d 22 77 6f 74 2d 76 65 72 69 66 69 63 61 74
 [401] 69 6f 6e 22 20 63 6f 6e 74 65 6e 74 3d 22 36 38 32 35 39 37 62 36 31 63 32 31 38 37 33 35 34 35 63 61 22 2f 3e 0a 3c 6d
 [441] 65 74 61 20 6e 61 6d 65 3d 22 67 6f 6f 67 6c 65 2d 73 69 74 65 2d 76 65 72 69 66 69 63 61 74 69 6f 6e 22 20 63 6f 6e 74
 [481] 65 6e 74 3d 22 6c 74 73 4c 4a 58 57 6b 74 6c 78 4b 48 70 54 77 38 78 4b 6f 48 73 46 32 4d 6e 41 6a 56 39 6f 37 4f 30 46
 [521] 6d 74 47 6b 31 63 5f 59 22 20 2f 3e 0a 3c 74 69 74 6c 65 3e 72 75 64 2e 69 73 3c 2f 74 69 74 6c 65 3e 0a 3c 6c 69 6e 6b
 [561] 20 72 65 6c 3d 22 73 68 6f 72 74 63 75 74 20 69 63 6f 6e 22 20 68 72 65 66 3d 22 68 74 74 70 73 3a 2f 2f 72 75 64 2e 69
 [601] 73 2f 66 61 76 69 63 6f 6e 2e 69 63 6f 22 20 74 79 70 65 3d 22 69 6d 61 67 65 2f 76 6e 64 2e 6d 69 63 72 6f 73 6f 66 74
 [641] 2e 69 63 6f 6e 22 20 2f 3e 0a 3c 6c 69 6e 6b 20 72 65 6c 3d 22 69 63 6f 6e 22 20 68 72 65 66 3d 22 68 74 74 70 73 3a 2f
 [681] 2f 72 75 64 2e 69 73 2f 66 61 76 69 63 6f 6e 2e 69 63 6f 22 20 74 79 70 65 3d 22 69 6d 61 67 65 2f 67 69 66 22 20 2f 3e
 [721] 0a 3c 6c 69 6e 6b 20 72 65 6c 3d 27 69 6e 64 65 78 27 20 74 69 74 6c 65 3d 27 72 75 64 2e 69 73 27 20 68 72 65 66 3d 27
 [761] 68 74 74 70 73 3a 2f 2f 72 75 64 2e 69 73 2f 27 20 2f 3e 0a 3c 73 63 72 69 70 74 20 73 72 63 3d 22 68 74 74 70 73 3a 2f
 [801] 2f 75 73 65 2e 74 79 70 65 6b 69 74 2e 6e 65 74 2f 6d 6b 6e 38 69 69 77 2e 6a 73 22 3e 3c 2f 73 63 72 69 70 74 3e 0a 3c
 [841] 73 63 72 69 70 74 3e 74 72 79 7b 54 79 70 65 6b 69 74 2e 6c 6f 61 64 28 7b 20 61 73 79 6e 63 3a 20 74 72 75 65 20 7d 29
 [881] 3b 7d 63 61 74 63 68 28 65 29 7b 7d 3c 2f 73 63 72 69 70 74 3e 0a 3c 6c 69 6e 6b 20 72 65 6c 3d 22 73 74 79 6c 65 73 68
 [921] 65 65 74 22 20 68 72 65 66 3d 22 73 74 79 6c 65 2e 63 73 73 22 20 74 79 70 65 3d 22 74 65 78 74 2f 63 73 73 22 20 2f 3e
 [961] 0a 3c 6d 65 74 61 20 6e 61 6d 65 3d 22 64 65 73 63 72 69 70 74 69 6f 6e 22 20 63 6f 6e 74 65 6e 74 3d 22 59 6f 75 20 61
 [ reached getOption("max.print") -- omitted 3532 entries ]
jay commented 7 years ago

What are the version strings from both versions? Can you test the older version to see if it still works?

jeroen commented 7 years ago

@jay the thing is that we switched from OpenSSL to WinSSL in the most recent version. So that's probably why this problem now suddenly appears.

jay commented 7 years ago

we switched from OpenSSL to WinSSL in the most recent version

ah ok. The reporter can try accessing the website in Internet Explorer, which also uses schannel/WinSSL.

maalsol commented 7 years ago

The site works fine if I access it using IE.

jeroen commented 7 years ago

@maalsol that's very strange. There must be something odd in the server config. Some questions:

certs <- openssl::download_ssl_cert("hostname")
str(certs)
maalsol commented 7 years ago 
*  Hostname in DNS cache was stale, zapped
*  Trying <IP>...
*  TCP_NODELAY set
*  Connected to <site> (<IP>) port 443 (#23)
*  schannel: SSL/TLS connection with  <site> port 443 (step 1/3)
*  schannel: disabled server certificate revocation checks
*  schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
*  schannel: sending initial handshake data: sending 164 bytes...
*  schannel: sent initial handshake data: sent 164 bytes
*  schannel: SSL/TLS connection with <site> port 443 (step 2/3)
*  schannel: failed to receive handshake, need more data
*  schannel: SSL/TLS connection with <site> port 443 (step 2/3)
*  schannel: encrypted data got 2798
*  schannel: encrypted data buffer: offset 2798 length 4096
*  schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid
*  Closing connection 11
*  schannel: shutting down SSL/TLS connection with <site> port 443
*  schannel: clear security context handle
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid

As a work around, is there any configuration to force curl (through httr) to use openSSL instead of winSSL and behave as it was prior to 3.0?

jeroen commented 7 years ago

@maalsol I could add such an option but actually I really want to get to the bottom of this. If you can't give us details about your certs, can you give us information on your version of IIS and TLS that the server is using?

This seems like a bug in libcurl @jay @bagder

dscho commented 7 years ago

is there any configuration to force curl (through httr) to use openSSL instead of winSSL

If cURL was compiled with both backends, you can set the environment variable CURL_SSL_BACKEND=openssl to force OpenSSL to be used.

jeroen commented 7 years ago

If cURL was compiled with both backends, you can set the environment variable CURL_SSL_BACKEND=openssl to force OpenSSL to be used.

That probably won't work, I have hardcoded this in C. Do you have any guess to what this error message means though?

dscho commented 7 years ago

If cURL was compiled with both backends, you can set the environment variable CURL_SSL_BACKEND=openssl to force OpenSSL to be used.

Note: I consider this a work-around, and would much rather see the original SChannel issue resolved, but I have to admit that I never saw anything like the error message described here.

dscho commented 7 years ago

The error message looks a little bit like the problem described in https://github.com/curl/curl/issues/983, even if that was with WINE, not with real Windows.

Keep in mind that I am far from an expert when it comes to Secure Channel. I am merely a user.

Having said that, one tid-bit looks interesting, though: it seems that the problem was while initializing the security context without ALPN (and it seems that ALPN is used by cURL only with Windows >8), it wanted NULL to be passed as inputbuf parameter. That is the case in the first call to InitializeSecurityContext(). However, when I look at the second call to said function, I do not see any use_alpn being mentioned.

Now, I wonder whether there is a chance for you to perform a really quick-and-dirty test where you recompile cURL simply with the second call changed such that it uses (BACKEND->use_alpn ? &inbuf_desc : NULL) instead of &inbuf_desc?

dscho commented 7 years ago

I wonder whether there is a chance for you to perform a really quick-and-dirty test where you recompile cURL simply with the second call changed such that it uses (BACKEND->use_alpn ? &inbuf_desc : NULL) instead of &inbuf_desc?

Please note that I have not the slightest idea whether that is even proper at this point in the code. I really only had a very quick look. But if you could test that, and if it fixes the error message, we would be a few steps further to a real fix.

jeroen commented 7 years ago

OK that might be worth looking into. However @maalsol mentions he uses a win7 client. @maalsol is there any chance you can test if the same problem appears on Windows 8 or Windows 10?

jay commented 7 years ago

Having said that, one tid-bit looks interesting, though: it seems that the problem was while initializing the security context without ALPN (and it seems that ALPN is used by cURL only with Windows >8), it wanted NULL to be passed as inputbuf parameter. That is the case in the first call to InitializeSecurityContext(). However, when I look at the second call to said function, I do not see any use_alpn being mentioned.

Please note that I have not the slightest idea whether that is even proper at this point in the code. I really only had a very quick look. But if you could test that, and if it fixes the error message, we would be a few steps further to a real fix.

The second call is to handle handshake data received from the server. In other words the inbuf on the second call to initialize the context is received data not to-be-sent data. It's part of the server's handshake to us. We can't mess with that data afaik.

Googling this I ran into a report I forgot I made at microsoft's forums which was never resolved. Basically some servers could not be connected to with TLS 1.2. Schannel I think must have more stringent certificate requirements for TLS 1.2. I'm not sure if it's related, but the reporter could try connecting with just TLS1.1 or TLS 1.0 (specify just one) and see if either of those work. In curl it looks like curl --tlsv1.0 or curl --tlsv1.1 . In rcurl I think you would use sslversion = but I don't know of the tls names

jeroen commented 7 years ago

@maalsol can you try if any of these change anything:

httr::set_config(config(ssl_version = 4))
httr::set_config(config(ssl_version = 5))
httr::set_config(config(ssl_version = 6))

Here is the full table with values:

> subset(curl_symbols, grepl("SSLVERSION_", curl_symbols$name ), select = c("name", "value"))
                           name  value
104     CURL_SSLVERSION_DEFAULT      0
111       CURL_SSLVERSION_SSLv2      2
112       CURL_SSLVERSION_SSLv3      3
113       CURL_SSLVERSION_TLSv1      1
114     CURL_SSLVERSION_TLSv1_0      4
115     CURL_SSLVERSION_TLSv1_1      5
116     CURL_SSLVERSION_TLSv1_2      6
117     CURL_SSLVERSION_TLSv1_3      7
jeroen commented 7 years ago

@maalsol do you have a chance to test if setting ssl_version has any effect? Also if the problem exists on Windows 8 / Windows 10 clients?

weshinsley commented 6 years ago

We have this problem (or one very similar) accessing a vault server using a non self signed certificate (this is with @richfitz at imperial). Our client machine is windows 10 (see below for full sessionInfo)

> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

(verbose output is identical)

We tried with the different ssl versions (using sslversion rather than ssl_version as that's what curl_options seems to suggest):

> curl_options(sslversion =4)
Error in curl_options(sslversion = 4) : unused argument (sslversion = 4)
> httr::set_config(config(sslversion =4))
> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_UNSUPPORTED_FUNCTION (0x80090302) - The function requested is not supported
> httr::set_config(config(sslversion =5))
> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_UNSUPPORTED_FUNCTION (0x80090302) - The function requested is not supported
> httr::set_config(config(sslversion =6))
> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

The site loads fine with chrome and firefox, but with IE we did get a "check this certificate" page. That seems not to be reproducible though as now it is claiming to load the page.

The site is on our private network so we can't use the ssldecoder. The output of openssl's download_ssl_cert is:

> certs <- openssl::download_ssl_cert("support.montagu.dide.ic.ac.uk", port = 8200)
> certs
[[1]]
[x509 certificate] support.montagu.dide.ic.ac.uk
md5: d0ab44375961a8f41bd850376f33d0d1
sha1: eecec2a2e2fde61d5d8cf2de56b18ebbb84f3e27

[[2]]
[x509 certificate] QuoVadis Global SSL ICA G3
md5: 5991d56bed59492776acb9eb100c7576
sha1: e90bcca3d134127ef646e854723f137d7971db64

(The certificate is issued by central IT here and they provide an intermediate certificate too).

With other suggestions further up the thread:

Setting ssl_options =1 does not help

> httr::set_config(config(ssl_options =1))
> httr::set_config(config(sslversion =0))
> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

This worked with the previous release of curl on this same computer, and works on non-Windows clients.

R version 3.3.3 (2017-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] httr_1.3.1 curl_3.1  

loaded via a namespace (and not attached):
[1] R6_2.2.2      tools_3.3.3   yaml_2.1.14   openssl_0.9.7
jeroen commented 6 years ago

@jay I think this may be a bug in libcurl winssl? Can you repreduce the issue using the server from weshinsley above ?

jay commented 6 years ago

@jeroen I can't access that website:

curl: (7) Failed to connect to support.montagu.dide.ic.ac.uk port 8200: Timed out

He could download the server certificate and run certutil /verify on it for diagnostic output. And if there's a way to disable the revocation check in R he could try that. In libcurl it looks like 

curl_easy_setopt(curl, CURLOPT_SSL_OPTIONS, CURLSSLOPT_NO_REVOKE);
weshinsley commented 6 years ago

support.montagu.dide.ic.ac.uk:8200 (vault) is behind our firewall, so not available from outside Imperial. We use the same certificate for everything on that machine, and we also run a https web server on port 10443:-

> httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.
> httr::GET("https://support.montagu.dide.ic.ac.uk:10443")
Response [https://support.montagu.dide.ic.ac.uk:10443/]
  Date: 2018-01-23 11:34
  Status: 200
  Content-Type: text/html
  Size: 2.72 kB
<!DOCTYPE html>
<html>
    <head>
        <title>Vaccine Impact Modelling Consortium - Montagu</title>
        <link rel="stylesheet" href="./resources/style.css" />
    </head>
    <body>

        <div id="content">
            <table class="header">
...

I downloaded the certificate through a browser and used certutil /verify:

Issuer:
    CN=QuoVadis Global SSL ICA G3
    O=QuoVadis Limited
    C=BM
  Name Hash(sha1): 6aae0d71a907ce6237901e87ed4c8dfa97a207d2
  Name Hash(md5): dbb46264dee5c19ca40b21619f8f41bc
Subject:
    CN=support.montagu.dide.ic.ac.uk
    OU=Dept of Infectious Disease Epidemiology
    O=Imperial College of Science, Technology and Medicine
    L=LONDON
    S=Greater London
    C=GB
  Name Hash(sha1): 4380558b0793209ef1bb9692ade83ba3c7b51830
  Name Hash(md5): 9e052181ade8faf3b10754a4f74dbcc9
Cert Serial Number: 551c37bdc67f171caaa3d204dabc438825ad8197

dwFlags = CA_VERIFY_FLAGS_CONSOLE_TRACE (0x20000000)
dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000)
ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000)
HCCE_LOCAL_MACHINE
CERT_CHAIN_POLICY_BASE
-------- CERT_CHAIN_CONTEXT --------
ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
ChainContext.dwRevocationFreshnessTime: 23 Hours, 40 Minutes, 56 Seconds

SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
SimpleChain.dwRevocationFreshnessTime: 23 Hours, 40 Minutes, 56 Seconds

CertContext[0][0]: dwInfoStatus=102 dwErrorStatus=0
  Issuer: CN=QuoVadis Global SSL ICA G3, O=QuoVadis Limited, C=BM
  NotBefore: 06/07/2017 12:12
  NotAfter: 06/07/2020 12:12
  Subject: CN=support.montagu.dide.ic.ac.uk, OU=Dept of Infectious Disease Epidemiology, O="Imperial College of Science, Technology and Medicine", L=LONDON, S=Greater London, C=GB
  Serial: 551c37bdc67f171caaa3d204dabc438825ad8197
  SubjectAltName: DNS Name=support.montagu.dide.ic.ac.uk
  Cert: eecec2a2e2fde61d5d8cf2de56b18ebbb84f3e27
  Element.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)
  Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
    CRL (null):
    Issuer: CN=QuoVadis OCSP Authority Signature, OU=OCSP Responder, O=QuoVadis Limited, C=BM
    ThisUpdate: 22/01/2018 11:57
    NextUpdate: 24/01/2018 11:57
    CRL: feecbe467100593adee25ab6036a5f0a3de31f9a
  Issuance[0] = 1.3.6.1.4.1.8024.0.2.100.1.1 
  Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication
  Application[1] = 1.3.6.1.5.5.7.3.2 Client Authentication

CertContext[0][1]: dwInfoStatus=102 dwErrorStatus=0
  Issuer: CN=QuoVadis Root CA 2 G3, O=QuoVadis Limited, C=BM
  NotBefore: 06/11/2012 14:50
  NotAfter: 06/11/2022 14:50
  Subject: CN=QuoVadis Global SSL ICA G3, O=QuoVadis Limited, C=BM
  Serial: 7ed6e79cc9ad81c4c8193ef95d4428770e341317
  Cert: e90bcca3d134127ef646e854723f137d7971db64
  Element.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)
  Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
    CRL (null):
    Issuer: CN=QuoVadis OCSP Authority Signature, OU=OCSP Responder, O=QuoVadis Limited, C=BM
    ThisUpdate: 22/01/2018 11:57
    NextUpdate: 24/01/2018 11:57
    CRL: fd5b0324e23ddbf702fb6f756305e1ffa516a185
  Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication
  Application[1] = 1.3.6.1.5.5.7.3.2 Client Authentication
  Application[2] = 1.3.6.1.5.5.7.3.4 Secure Email
  Application[3] = 1.3.6.1.5.5.7.3.3 Code Signing
  Application[4] = 1.3.6.1.5.5.7.3.8 Time Stamping
  Application[5] = 1.3.6.1.5.5.7.3.9 OCSP Signing

CertContext[0][2]: dwInfoStatus=10c dwErrorStatus=0
  Issuer: CN=QuoVadis Root CA 2 G3, O=QuoVadis Limited, C=BM
  NotBefore: 12/01/2012 18:59
  NotAfter: 12/01/2042 18:59
  Subject: CN=QuoVadis Root CA 2 G3, O=QuoVadis Limited, C=BM
  Serial: 445734245b81899b35f2ceb82b3b5ba726f07528
  Cert: 093c61f38b8bdc7d55df7538020500e125f5c836
  Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)
  Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)
  Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
  Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication
  Application[1] = 1.3.6.1.5.5.7.3.2 Client Authentication
  Application[2] = 1.3.6.1.5.5.7.3.4 Secure Email
  Application[3] = 1.3.6.1.5.5.7.3.3 Code Signing
  Application[4] = 1.3.6.1.5.5.7.3.8 Time Stamping
  Application[5] = 1.3.6.1.5.5.7.3.9 OCSP Signing
  EV[0] = 1.3.6.1.4.1.8024.0.2.100.1.2 

Exclude leaf cert:
  Chain: e00d756d7ad927dff123eb085f04423543f8d0b8
Full chain:
  Chain: 4a9688d43ec5344c9e3c1e7d87395def981b2cb4
------------------------------------
Verified Issuance Policies:
    1.3.6.1.4.1.8024.0.2.100.1.1
Verified Application Policies:
    1.3.6.1.5.5.7.3.1 Server Authentication
    1.3.6.1.5.5.7.3.2 Client Authentication
Leaf certificate revocation check passed
CertUtil: -verify command completed successfully.

Disabling verification isn't a live option here, as we're wanting to securely retrieve secrets from the vault.

jeroen commented 6 years ago

He doesn't suggest to disable verification but disable the revocation check. Not as a permanent solution but to narrow down where the problem appears. Could you try this?

jeroen commented 6 years ago

It's strange that the same cert works fine on the other port. Is that port running a different httpd? What version of IIS/windows is hosting the problematic port?

richfitz commented 6 years ago

Hi @jeroen - the

It's strange that the same cert works fine on the other port. Is that port running a different httpd? What version of IIS/windows is hosting the problematic port?

The port that works is running nginx (in a docker container). The one that doesn't is running vault. Dockerfile is here - you can see how we're dealing with the intermediate certificate there

What is the (r) curl option for skipping the revocation check:

grep("revoke", names(curl::curl_options()), value = TRUE)
# character(0)

It does look like this might already be happening though? https://github.com/jeroen/curl/blob/a5a65aac83d90dc45132c769030254bb9d5bb1c7/src/handle.c#L132

richfitz commented 6 years ago

(but yeah, the two ports run totally different servers, sharing no code at all - but do have the same certificate)

jeroen commented 6 years ago

Ok when I'm back home I'll try to reproduce this using the docker container and my own cert.

jeroen commented 6 years ago

I can't test this on my phone but could it be that the certs are concatinated in the wrong order? This is a common issue that trips over some but not all clients.

richfitz commented 6 years ago

I'll give that a shot

richfitz commented 6 years ago

I tried doing that but the vault server will not start (with error:

Error initializing listener of type tcp: error loading TLS cert: tls: private key does not match public key

-- nothing to do with curl; this is all on the server side) and then dies. So the certs are presumably in the right order?

richfitz commented 6 years ago

And one further piece that may or may not be useful. From a linux machine with curl 7.47.0:

$ curl --verbose https://support.montagu.dide.ic.ac.uk:8200/v1/sys/init
*   Trying 129.31.26.30...
* Connected to support.montagu.dide.ic.ac.uk (129.31.26.30) port 8200 (#0)
* found 149 certificates in /etc/ssl/certs/ca-certificates.crt
* found 608 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: support.montagu.dide.ic.ac.uk (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: C=GB,ST=Greater London,L=LONDON,O=Imperial College of Science\, Technology and Medicine,OU=Dept of Infectious Disease Epidemiology,CN=support.montagu.dide.ic.ac.uk
*    start date: Thu, 06 Jul 2017 12:12:07 GMT
*    expire date: Mon, 06 Jul 2020 12:12:03 GMT
*    issuer: C=BM,O=QuoVadis Limited,CN=QuoVadis Global SSL ICA G3
*    compression: NULL
* ALPN, server accepted to use http/1.1
> GET /v1/sys/init HTTP/1.1
> Host: support.montagu.dide.ic.ac.uk:8200
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: no-store
< Content-Type: application/json
< Date: Wed, 24 Jan 2018 12:11:38 GMT
< Content-Length: 21
< 
{"initialized":true}
* Connection #0 to host support.montagu.dide.ic.ac.uk left intact

That might give some hints as to how the certificates are being passed back perhaps?

jay commented 6 years ago

Can you do a wireshark capture of the failed connection in r without setting sslversion? Open wireshark and capture using filter host support.montagu.dide.ic.ac.uk and port 8200 and it will capture only traffic to/from there. Then try the connection. Then stop the packet capture and File>Save as pcap and then zip that up and upload it here. It will contain your MAC address and local IP so if that's a problem you can use something like TraceWrangler to sanitize it.

weshinsley commented 6 years ago

Ok, I've captured that - we're behind a firewall, so I don't think there's much risk - but can you download it from https://mrcdata.dide.ic.ac.uk/resources/test8200.zip, then I'll remove it when you've got it...

jay commented 6 years ago

got it. I anonymized it with tracewrangler if you want to delete the original. anonymized copy is attached test8200_anon.zip

jay commented 6 years ago

It shows the server rejecting a certificate as bad certificate

Frame 10: 1344 bytes on wire (10752 bits), 1344 bytes captured (10752 bits) on interface 0
Ethernet II, Src: f2:27:af:3c:ff:6d (f2:27:af:3c:ff:6d), Dst: f2:59:57:80:82:3c (f2:59:57:80:82:3c)
Internet Protocol Version 4, Src: 10.10.10.10, Dst: 129.31.26.30
Transmission Control Protocol, Src Port: 50818, Dst Port: 8200, Seq: 190, Ack: 3601, Len: 1290
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 1234
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 929
            Certificates Length: 926
            Certificates (926 bytes)
                Certificate Length: 923
                Certificate: 308203973082027fa003020102020a144637880fe62ecce8... (id-at-commonName=w.hinsley@imperial.ac.uk)
                    signedCertificate
                        version: v3 (2)
                        serialNumber: 0x144637880fe62ecce830
                        signature (sha256WithRSAEncryption)
                        issuer: rdnSequence (0)
                        validity
                        subject: rdnSequence (0)
                        subjectPublicKeyInfo
                        extensions: 5 items
                    algorithmIdentifier (sha256WithRSAEncryption)
                    Padding: 0
                    encrypted: aca1248e35711b7c820165bd68ed02c790bc59399dec7429...
        Handshake Protocol: Client Key Exchange
            Handshake Type: Client Key Exchange (16)
            Length: 33
            EC Diffie-Hellman Client Params
        Handshake Protocol: Certificate Verify
            Handshake Type: Certificate Verify (15)
            Length: 260
            Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                Signature Hash Algorithm Hash: SHA256 (4)
                Signature Hash Algorithm Signature: RSA (1)
            Signature length: 256
            Signature: 3fd37558f415c2dc90d476e6c38e8f813b3b959674ab0797...
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
        Content Type: Change Cipher Spec (20)
        Version: TLS 1.2 (0x0303)
        Length: 1
        Change Cipher Spec Message
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 40
        Handshake Protocol: Encrypted Handshake Message

to see this for yourself you have to id port 8200 as SSL, Edit > Preferences > HTTP capture3 capture

weshinsley commented 6 years ago

So we get a bad certificate error if (on my Win 10 desktop) I do: httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init") - as above.

If I do the same thing on a Win 2012R2 server, it's successful - it returns the 404 we intend, rather than throwing the exception:-

httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init") Response [https://support.montagu.dide.ic.ac.uk:8200/sys/init] Date: 2018-01-24 21:41 Status: 404 Content-Type: text/plain; charset=utf-8 Size: 19 B 404 page not found

Here's the wireshark for that: test8200_win2012r2.zip

We also get sucess back on my Win 10 desktop, if I go for the web server on support - it's the same certificate though...

httr::GET("https://support.montagu.dide.ic.ac.uk:10443") test10443.zip

jay commented 6 years ago

I think you anonymized it too much, I can't decode the packets as SSL. Remove unknown layers may be checked and also rand destination IP but you should leave both of those unmodified unchecked. When you leave that info it's not fully anonymous of course but it's needed for a correct read of the session.

weshinsley commented 6 years ago

Ah, sorry - these tools are a bit new to me, and I had left all the defaults set... are these better?

test8200_win2012r2.zip test10443.zip

jay commented 6 years ago

Yeah they're better but they include the unmodified source IP, I'm not sure how important that is to you. The MACs are randomized though.

If I do the same thing on a Win 2012R2 server, it's successful - it returns the 404 we intend, rather than throwing the exception:-

Do you mean if you use the same version of rcurl on the other computer win2012r2 then it's successful? Because it appears to me that a different SSL library is used (OpenSSL?) therefore it's not the same.

In test8200_win2012r2 I can see no certificate is sent by the client in response to the server's certificate request which is probably why things go ok.

    TLSv1.2 Record Layer: Handshake Protocol: Certificate
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 7
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 3
            Certificates Length: 0

putting that aside and getting back to schannel we have some code that supposedly allows continuing on an optional client certificate:

https://github.com/curl/curl/blob/d6c21c8eec597a925d2b647cff3d58ac69de01a0/lib/vtls/schannel.c#L672-L681

according to microsoft:

_"When the server requests client authentication, the client must send the server one of its certificates. By default, Schannel will, with no notification to the client, attempt to locate a client certificate and send it to the server. To disable this feature, clients specify ISC_REQ_USE_SUPPLIED_CREDS when calling the InitializeSecurityContext (Schannel) function. When this flag is specified, Schannel will return SEC_I_INCOMPLETECREDENTIALS to the client when the server requests authentication and the client has not previously supplied a certificate."

Can you turn on verbose mode, do you see schannel: a client certificate has been requested. I'm wondering if our code is wrong or this is to be expected since the certificate is bad (supposedly.. did you check that?)

weshinsley commented 6 years ago

I've upgraded the Win2012R2 to R 3.3.3, and installed.packages() now says curl is version 3.3.3 - which matches the Win 10 desktop. I still get a "successful" 404, as before...

test8200_r333_win2012r2.zip

So, as far as R installations go, they are as identical as I can make them. I don't know how this relates to the underlying windows version though...

Running with verbose on Win 10, afraid this is all I get:-

httr::with_verbose(httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")) Error in curl::curl_fetch_memory(url, handle = handle) : schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

On Win 2012 R2: httr::with_verbose(httr::GET("https://support.montagu.dide.ic.ac.uk:8200/sys/init")) -> GET /sys/init HTTP/1.1 -> Host: support.montagu.dide.ic.ac.uk:8200 -> User-Agent: libcurl/7.56.1 r-curl/3.1 httr/1.3.1 -> Accept-Encoding: gzip, deflate -> Accept: application/json, text/xml, application/xml, / -> <- HTTP/1.1 404 Not Found <- Cache-Control: no-store <- Content-Type: text/plain; charset=utf-8 <- X-Content-Type-Options: nosniff <- Date: Wed, 24 Jan 2018 23:08:34 GMT <- Content-Length: 19 <- Response [https://support.montagu.dide.ic.ac.uk:8200/sys/init] Date: 2018-01-24 23:08 Status: 404 Content-Type: text/plain; charset=utf-8 Size: 19 B 404 page not found

As far as the certificate goes: does the certutil /verify a few posts above reveal anything? We have no (other) reason to think it is bad - it seems to be perfectly valid in all the browsers, and using vault on the older Win versions/linux seems fine too.

jay commented 6 years ago

So, as far as R installations go, they are as identical as I can make them. I don't know how this relates to the underlying windows version though...

What I was saying was you had one client in Windows 10 using rcurl w/schannel and another client in Windows 2012 R2 using rcurl w/ probably a different SSL library.

test8200_r333_win2012r2.zip

That is missing the ssl handshake.

As far as the certificate goes: does the certutil /verify a few posts above reveal anything? We have no (other) reason to think it is bad - it seems to be perfectly valid in all the browsers, and using vault on the older Win versions/linux seems fine too.

The certutil is for the server certificate which looks fine. The issue here is the client certificate (wes.cer.zip) that schannel sends to the vault server, which is rejected. The rejection is fatal and the handshake cannot be continued. That is the server's decision. What is unclear to me is whether or not the other connections you are making that are successful are also sending the client certificate, or if it is just schannel, and why the server is rejecting it.

weshinsley commented 6 years ago

Thanks for your patience in explaining this - it's not an area I know much about. Sounds like Win 2012R2 / Win10 must be using different SSL libraries then; one tries to do a handshake with a client certificate and the other appears not to. And also something happens differently about that interaction when connecting to vault on 8200, than to nginx on 10443. I don't know how you find out which SSL library is being used deep down...

The client certificate is not one I've "consciously" made or configured anywhere - I'm not sure what "lyncpool.imperial" is/does as the issuer - when I open the certificate and look at the Certification Path, it says "The issuer of this certificate could not be found." - might be reasonably interpreted as a "bad" certificate? I can talk to IT here about that, to try and work out what the story is.

jeroen commented 6 years ago

The R curl client is configured to use the winssl backend on Windows 7 and up (see code). On Windows Vista/2008 it will fall back on OpenSSL but I don't think that is the case here?

@weshinsley can you check that curl::curl_version()$ssl_version is the same on both your win10 and win2k12 clients?

weshinsley commented 6 years ago

Confirmed on both:

curl::curl_version()$ssl_version [1] "(OpenSSL/1.1.0f) WinSSL"

weshinsley commented 6 years ago

Just to throw a spanner in the works... I thought I'd build a fresh Windows 10 laptop from scratch - updated to v1709. (My dekstop is v1703 - the latest update isn't approved for us in department yet) - installed R 3.3.3, httr package - same ssl version reported above... and I get the 404 message expected, not the SEC_E_CERT_UNKNOWN.

So, it looks like it's not Win10 generally that's the problem, but something has gone wrong on my Win 10 desktop/its configuration, perhaps certificate-related, that makes the difference.