Closed niemyjski closed 4 years ago
Possibly references:
cc: @stephentoub
Yes, to use several of the HttpClient features related to certificates, like ServerCertificateCustomValidationCallback, you need to be using a version of libcurl built against OpenSSL; the version you're using is the one that comes with macOS, built against the SecureTransport backend. We do not have the ability to plug in to that backend, hence we throw an exception to indicate that the callback won't be used.
@stephentoub I don't think this acceptable. I couldn't make any http request because of it and I'm using the install instructions on the main website which many many others are going to use (that are on osx). Could we get a check to see if it's supported like other handler features? Any request without validation callback is better than nothing.. I get why you are throwing it.. but most peoples http client code is going to be try catched and they are not going to see this...
I'm using the install instructions on the main website
We can look to augment the instructions / release notes with related details. cc: @leecow
Any request without validation callback is better than nothing
I'm not understanding this part. If you want to avoid the error, you can avoid hooking up the callback. And if you've specified a callback, not invoking it could be a significant security hole, as we'd end up using the default validation of a security certificate rather than the validation the developer has explicitly requested via the callback.
but most peoples http client code is going to be try catched and they are not going to see this
I don't understand this part either. Can you elaborate?
Could we get a check to see if it's supported like other handler features?
You mean a capability property, like IsCustomServerCertificateValidationSupported? That's a reasonable request for the future. Thanks.
cc: @ericeil, @bartonjs
Yeah, I get all of your points and agree when it comes to security.
if (handler.SupportsServerCertificateValidationCallback) {}
@stephentoub @bartonjs feel free to submit a PR to update the release notes / docs to make this more clear.
How is this supposed to work on OSX? Install some package on OSX? Or there is no way to make it work on OSX?
I don't think there is any way for this to work on OSX currently (at least I haven't found a way)
I don't think there is any way for this to work on OSX currently (at least I haven't found a way)
I'm sure there are other ways, but I did:
brew install curl --with-openssl
brew link --force curl
and then with that library being used:
export DYLD_LIBRARY_PATH=/usr/local/lib/
I can successfully use these features...
stoubmac:~ stoub$ cd testapp
stoubmac:testapp stoub$ cat Program.cs
using System;
using System.Net.Http;
namespace ConsoleApplication
{
public class Program
{
public static void Main(string[] args)
{
var handler = new HttpClientHandler { CheckCertificateRevocationList = true };
var http = new HttpClient(handler);
Console.WriteLine(http.GetAsync("https://www.bing.com").GetAwaiter().GetResult());
}
}
}
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Unhandled Exception: System.PlatformNotSupportedException: The libcurl library in use (7.43.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.
at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at ConsoleApplication.Program.Main(String[] args)
stoubmac:testapp stoub$ export DYLD_LIBRARY_PATH=/usr/local/lib/
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Cache-Control: max-age=0, private
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
Set-Cookie: SRCHD=AF=NOFORM; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
Set-Cookie: SRCHUID=V=2&GUID=05A7870E43E942F68BAC35BB8A4C52C6; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
Set-Cookie: SRCHUSR=DOB=20160721; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
Set-Cookie: _SS=SID=0A99E65098CC6D950303EF09991F6C8C; domain=.bing.com; path=/
Set-Cookie: _EDGE_S=F=1&SID=0A99E65098CC6D950303EF09991F6C8C; path=/; httponly; domain=bing.com
Set-Cookie: _EDGE_V=1; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
Set-Cookie: MUID=3441EE6881166AB927F3E73180C56B38; path=/; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
Set-Cookie: MUIDB=3441EE6881166AB927F3E73180C56B38; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT
X-MSEdge-Ref: Ref A: 8C58C90CF2764DCC8D6E5F4EBDDA771C Ref B: 3F2B603999D10209F1CECA268112300C Ref C: Thu Jul 21 16:49:04 2016 PST
Date: Thu, 21 Jul 2016 23:49:04 GMT
Content-Type: text/html; charset=utf-8
}
stoubmac:testapp stoub$
Fantastic, the export DYLD_LIBRARY_PATH=/usr/local/lib/ was what i was missing! Thanks
What's the fix for CentOS?
@jchannon CentOS 6 had a curl-openssl package which would (have) work(ed). The problem is that the default libcurl package on CentOS uses Mozilla NSS as the TLS provider, and we don't have the ability to interpret their certificate pointers.
Normal CentOS curl:
$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Normal Ubuntu curl:
$ curl --version
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Note the Ubuntu one in this example is OpenSSL/1.0.1f
, whereas the CentOS one uses NSS/3.19.1 Basic ECC
.
Since curl doesn't have their own abstraction of certificates we would have to have support for each of the (currently 8) possible providers. Right now OpenSSL is the only one.
Since there's not a CentOS package for that anymore that I can find offhand; that might mean you have to build from source. Or maybe there's just something I'm missing in the CentOS package tree.
@bartonjs we have it working on CentOS now, what we did was rm -f /usr/bin/curl
, rm -f rm /lib64/libcurl*
, yum groupinstall "Development @Tools"
then downloaded latest curl tarball http://curl.haxx.se/download/curl-7.50.1.tar.gz
run ./configure, make, make install
However trying to debug my app on OSX I followed the above instructions with
brew install curl --with-openssl
brew link --force curl
export DYLD_LIBRARY_PATH=/usr/local/lib/
Now when I run dotnet build
I see this:
Unhandled Exception: System.TypeInitializationException: The type initializer for 'Crypto' threw an exception. ---> System.TypeInitializationException: The type initializer for 'CryptoInitializer' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'System.Security.Cryptography.Native': The specified module could not be found.
(Exception from HRESULT: 0x8007007E)
at Interop.CryptoInitializer.EnsureOpenSslInitialized()
at Interop.CryptoInitializer..cctor()
--- End of inner exception stack trace ---
at Interop.Crypto..cctor()
--- End of inner exception stack trace ---
at Interop.Crypto.GetRandomBytes(Byte* buf, Int32 num)
at System.IO.Path.GetCryptoRandomBytes(Byte* bytes, Int32 byteCount)
at System.IO.Path.GetRandomFileName()
at Microsoft.DotNet.InternalAbstractions.TemporaryDirectory..ctor()
at Microsoft.Extensions.EnvironmentAbstractions.DirectoryWrapper.CreateTemporaryDirectory()
at Microsoft.DotNet.Configurer.NuGetPackagesArchiver..ctor()
at Microsoft.DotNet.Cli.Program.ConfigureDotNetForFirstTimeUse(INuGetCacheSentinel nugetCacheSentinel)
at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
at Microsoft.DotNet.Cli.Program.Main(String[] args)
[1] 9392 abort dotnet build
I am less than amused ðŸ˜
Also these instructions are the complete opposite of what is shown on www.dot.net
brew update
brew install openssl
ln -s /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib /usr/local/lib/
ln -s /usr/local/opt/openssl/lib/libssl.1.0.0.dylib /usr/local/lib/
https://github.com/dotnet/cli/issues/3964 see this issue @jchannon
Thanks, the fix buried in there was to run sudo install_name_tool -add_rpath /usr/local/opt/openssl/lib /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0/System.Security.Cryptography.Native.dylib
Quite frankly this is shit show and MS need to sort this out.
Trying to debug a simple http request with osx now results in this error message
"One or more errors occurred. (The libcurl library in use (7.43.0) and its SSL backend (\"SecureTransport\") do not support custom handling of certificates. A libcurl built with OpenSSL is required.)"
The code is:
var handler = new HttpClientHandler { CookieContainer = new CookieContainer() };
handler.AllowAutoRedirect = false;
handler.ServerCertificateCustomValidationCallback = (msg, cert, chain, errors) => true;
var httpClient = new HttpClient(handler);
var resp = httpClient.PostAsync(url, new StringContent(JsonConvert.SerializeObject(loginDTO), Encoding.UTF8, "application/json")).Result;
Yet when I run curl --version
from terminal I get this
curl 7.50.1 (x86_64-apple-darwin15.6.0) libcurl/7.50.1 OpenSSL/1.0.2h zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets
The solution to this when debugging is to set the DYLD_LIBRARY_PATH
environment variable. To do this in VSCode you add this to launch.json
"env": {
"DYLD_LIBRARY_PATH":"/usr/local/lib/"
}
I agree, it would be great if someone could figure out an easy solution. MacOS 10.11.x (El Capitan) makes it harder to change the OpenSSL version that ships with OS. I am hoping that 10.12 (Sierra) has a version of OpenSSL that plays nicer with ASP.NET.
/usr/bin/openssl OpenSSL 0.9.8zh 14 Jan 2016
on MacOS 10.11 (El Capitan) Haven't had a chance to install 10.12 beta yet.
Also, I deploy my Asp.Net core web app to an Azure instance and than I see the error message of
The specified CGI application encountered an error and the server terminated the process.
With Core 1.1, I run into this issue again on Mac Sierra. Reinstalled with the instructions (OpenSSL 1.0.2j) and the symlinks. The dotnet new command runs fine, but when I use the httpclient:
(The libcurl library in use (7.51.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.) ---> System.PlatformNotSupportedException: The libcurl library in use (7.51.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.
I used the instructions in other reports to use a openssl-build of curl, and the command line reports doing so. But the dotnet project seems to reference the system version?
.NET Command Line Tools (1.0.0-preview2-1-003177)
Product Information:
Version: 1.0.0-preview2-1-003177
Commit SHA-1 hash: a2df9c2576
Runtime Environment:
OS Name: Mac OS X
OS Version: 10.12
OS Platform: Darwin
RID: osx.10.12-x64
libcurl.dylib -> ../Cellar/curl/7.52.1/lib/libcurl.dylib
When using DYLD_LIBRARY_PATH things mess up quickly (methods not found etc), and the rpath commands do not seem to help at all.
As I understood, things changed in core 1.1 to both support native Apple and OpenSSL. But how do I choose between them?
@hlogmans The 1.1 release started using Security.framework for hashing, HMAC, and symmetric encryption. Asymmetric cryptography, certificates, and TLS are all still performed by OpenSSL.
otool -L System.Net.Http.Native.dylib
should tell you how that library linked against curl (@rpath or a direct path reference), which should aid in tracking down why it isn't loading the build of the library you expect.
System.Net.Http.Native.dylib: @rpath/System.Net.Http.Native.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
/usr/lib/libcurl.4.dylib: /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0) /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57740.30.50) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1348.28.0) /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0) /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
So that seems quite logical that it targets the wrong one. Next step: how to fix it form me, and how to fix it for everyone... (and thanks for your support @bartonjs)
Update: sudo install_name_tool -change /usr/lib/libcurl.4.dylib /usr/local/opt/curl/lib/libcurl.4.dylib System.Net.Http.Native.dylib
@karelz The release docs needs to be updated as mentioned in this comment, can you please reroute to the right person.
cc @mairaw
Is it about the installation instructions on the dot.net site @Priya91?
@mairaw Yes..
Ok thanks for confirming @Priya91. @leecow @kendrahavens, can one of you take a look at this?
Thanks @mairaw, I'll look into the install instruction updates.
I am having the same issue with CentOS 7-x64 Azure. Do I have to create a new issue?
@deepumi Custom handling is only supported on Linux when libcurl uses openssl as its TLS backend. Feel free to open a CentOS-specific issue, though.
@deepumi Centos default curl installation comes with NSS as crypto backend. Please recompile curl with openssl and then it should work. You can follow the instructions here
@Priya91 Thanks! I have to work with our Network team on this.
In most cases what we are trying to do with ServerCertificateCustomValidationCallback is to disable server side certificate validation or overwrite the root certificate handling with custom root certificates, both something that is support on libcurl with SecureTransport.
Would it be possible to extend the interface of HttpClient to support this more directly fx.
httpHandler.ServerCertificateValidation = false; httpHandler.ServerCertificateRootCaFile = "/.../ca.pem";
Both options would be super easy to implement on platforms that support ServerCertificateCustomValidationCallback as this is very few lines of code and it would make life a lot easier on platforms where certificate validation is harder to overwrite.
@tlbdk I was of the impression that custom validation callbacks are not supported on OSX with SecureTransport. cc @bartonjs
@priya91 they are not when using the default system libcurl, but disabling server certificate validation and using a custom root ca list is, libcurl exposes API for this.
Completely disabling validation is something that we could support, if there were API to indicate that was the caller choice. Similarly, if we were to have an API which allowed the specification of a filename to use as a replacement root we could pass that through. (I certainly wouldn't expect us to write a temporary file on a caller's behalf)
What would be generally more useful is if https://github.com/curl/curl/issues/685 were implemented, so we could hook the handshake-complete callback and provide the same level of support that we have with libcurl+openssl on Linux (or, equivalently "the same level of support we have for directly using SslStream").
While I agree that it's unfortunate that custom handshake validation is not possible with HttpClientHandler on macOS, I'm not sure that the level of specific API that we'd need to plumb the options to libcurl make a whole lot of sense.
@bartonjs It would enable me to make my client certificate test work on all platforms without having to install certicates. And even if curl gets general support for this it will take a while before it will be shipped, so any way to do this now would be very much appropriate.
Is there a plan to address this in 2.0? This is a pretty big issue and it's been on going for about a year now.
It is currently tracked as documentation bug. Which we should address in 2.0.
I see several different asks on this thread. Some about MacOS, some about Linux. @niemyjski can you please explicitly describe what it is you would like to see? What are the customer scenarios? Is it "just" for easier testing? Which OS - Mac vs. Linux vs. both? Are there (potential) workaround? (like libcurl changes)
In general, regarding new APIs in 2.0 - it is kind of late -- see details in https://github.com/dotnet/corefx/issues/17619#issuecomment-296872132. While it is not unimaginable, it would have to have very strong case (high demand, no workarounds, low risk, etc.) - basically a DCR. Also note that networking v-team has lots of critical bugs to fix for 2.0. We are a bit stretched already in that area. Just finding someone available and capable of doing it (with high quality and expertise) would be a challenge at this moment.
It just needs to work, we have this code in our exceptionless nuget package which gets installed into thousands of apps. If you are running on linux or osx and we have this code in place you don't get any warning it just doesn't work (no http requests will be made). We can't relie on install documentation when this goes out to customers who then deploy on customers machines. We just need it to work.
@niemyjski it would help if you could answer all my questions - I am trying to help here, but we (or maybe just I) need summary of all relevant info to be able to make any decisions. The discussion above is a bit confusing to me :(.
Here's my attempt:
if (handler.SupportsServerCertificateValidationCallback) {}
. There are already other supports* properties and this makes sense. It allows us to define it if it's supported otherwise we don't.@niemyjski that sounds reasonable. Can you please file a new issue to track adding SupportsServerCertificateValidationCallback
? We should discuss it separately - I would love to see the use cases, discussion about workarounds and how common the scenario is. Please follow sample in API review process.
Note that even if we add it to .NET Core 2.0, it won't be available in .NET Standard 2.0.
@karelz Would be happy to do a pull request if we could extend the interface, I don't think this would take a lot of time to implement, fx for the "httpHandler.ServerCertificateValidation = false;" option, the code is already doing this when ServerCertificateValidationCallback is supported as it disables internal validation in libcurl. Getting it to work on other platforms where ServerCertificateValidationCallback is support is also simple as it just setting a delegate that returns true.
Is a bit more work and might be more controversial for the "httpHandler.ServerCertificateRootCaFile = "/.../ca.pem";" option as you would need to do actual certificate chain validation and also the certificates formats might differ from platform to platform.
So you would accept a pull request adding "httpHandler.ServerCertificateValidation = false;"?
For CentOS we still do the same as mentioned here - https://github.com/dotnet/corefx/issues/9728#issuecomment-239403404
For OSX we have to do the same as mentioned here - https://github.com/dotnet/corefx/issues/9728#issuecomment-239413686
It's a mess and needs sorting ASAP
@tlbdk sounds great, any chance you could open an issue to track this :). I thought we had created one before, but I guess it was just mentioned in this issue.
@tlbdk no guarantee if/when we take it - let's have the discussion in separate issue.
@jchannon "needs sorting ASAP" - what do you propose?
not having to jump through ridiculous hoops.
if i set the callback to true it should not validate the certs.
the issues seem to be with the underlying libraries however but i have no idea how you solve that but from a user perspective the api provided doesnt work
On 25 April 2017 at 22:59, Karel Zikmund notifications@github.com wrote:
@tlbdk https://github.com/tlbdk no guarantee if/when we take it - let's have the discussion in separate issue.
@jchannon https://github.com/jchannon "needs sorting ASAP" - what do you propose?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dotnet/corefx/issues/9728#issuecomment-297177693, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGapliAHYPwwxHdYrfzZ06AHZiYsUpFks5rzmzDgaJpZM4JAdiR .
@jchannon we can solve it by documentation. Or by making sure the exceptions are self-describing. Or by adding the API on the compat list. Do you have preference?
I just updated my Project to RTM and did testing on Windows and everything worked. I then went to test on OSX 10.11.5 (latest stable everything, dotnet, openssl) and ran into issues very quickly..
I'm getting
The libcurl library in use (7.43.0) and its SSL backend (\"SecureTransport\") do not support custom handling of certificates. A libcurl built with OpenSSL is required.
in stable latest everything on macOS...I narrowed it down to defining a custom validation callback breaks the http client.. I can't even do a
GET
.All the code can be found here: http://github.com/exceptionless/Exceptionless.Net