Add ConnectionPolicy.DisableSSLVerification for .NET Core

[ElasticCoder]

We are looking to build out a significant CosmosDB infrastructure and running scaled-out Change Feed Processors to process incoming orders.

To aid our developer experience, we intend to host the CosmosDB emulator on their development machine and run the Change Feed Clients within containers. We are developing using the .NET Core SDK and deploying to Alpine Linux containers using Docker for Windows. We may also use Service Fabric to orchestrate the change feed processors.

A welcomed move has been to enhance the emulator to support network connections other than 127.0.0.1. This now works, but unfortunately, the clients are still failing to connect because of the self-signed certificate running on the emulator.

It appears this is addressed for other languages other than .NET Core: This page indicates that “When connecting to the emulator from Python and Node.js SDKs, SSL verification is disabled.” The Python 2.1.0 SDK release “Added an option for disabling SSL verification when running against DocumentDB Emulator”.

With the latest .NET Core 1.3.2 version I do not have this option.

Will you add this option to the .NET Core SDK soon?

[kirankumarkolli]

@ElasticCoder did you try https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.servercertificatevalidationcallback(v=vs.110).aspx?

[ElasticCoder]

@kirankumarkolli Yes, when I last looked unfortunately ServicePointManager wasn't present in .NET Core.

[kirankumarkolli]

@ElasticCoder marking it as backlog & enhancement.

[joopscheer]

I'm having similar issues. My development environment is on macOS and the emulator runs on a Windows VM with network access. Before High Sierra I could add the self-signed certificate to the macOS certificate store. With High Sierra this no longer works. Any idea what the timeline is on this issue? In the meantime, have others fixed the certificate issue on macOS?

[MCrank]

I am having the same issue. Is there any update on when the ConnectionPolicy.DisableSSLVerification option will be included in .net core client? Doing any work in containers on a Windows machine with CosmosDB emulator running on localhost is pretty much moot without it.

[ElasticCoder]

I've repeated the tests again today, this time running the emulator in a VM and the client on the host laptop.

ServicePointManager doesn't work in .NET Core

The code below works if you compile as a normal .NET Console app, but fails if you compile it as a .NET Core app. The ServicePointManager callback is not called and SSL verification fails. ServicePointManager does not work for .NET core apps.

using System;
namespace Test
{
    class Program
    {
        static void Main(string[] args)
        {
            // This doesn't work in dotnet core
            System.Net.ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, errors) =>
            {
                Console.WriteLine("Ignoring SSL verification");
                return true;
            };

            try
            {
                var client = new System.Net.Http.HttpClient();
                var response = client.GetAsync("https://myVMname:8081/_explorer/index.html").GetAwaiter().GetResult();
                Console.WriteLine(response.StatusCode);
            }
            catch (Exception e)
            {
                while (e != null)
                {
                    Console.WriteLine(e.Message);
                    e = e.InnerException;
                }
            }
        }
    }
}

Importing the certificate doesn't help

I've followed other peoples articles by exporting the Self-Signed certificate from the emulator onto the client machine, but this doesn't work with .NET Core still. It still errors. I can see the SSL certificate provides localhost and as the subject alternative names, and have navigated using but this does not help. If you use a browser to navigate to the Data Explorer it still warns that the certificate is presented by localhost. You can accept the warning on a browser and navigate to the page, but any code will error.

The bottom line is that we cannot use ServicePointManager.ServerCertificateValidationCallback to bypass self-signed certificate verification for the emulator running over the network.

The benefit from bypassing SSL verification

Why is bypassing the certificate verification on the emulator so important to us? We are building high-density change feed clients that subscribe to Cosmos DB. These are currently deployed as Cloud Services but this is an incredibly wasteful use of compute and our strategy is to move to Kubernetes and AKS. Net Core is our strategic choice. We want to run our clients in containers and for the developers they should use the local emulator. (The last time we used a real Cosmos resource for Acceptance testing as part of our CI/CD process we ran up a bill of $15,000 in a month, which is not ideal). If our developers cannot connect to the emulator from a container then we either can't move to Kubernetes or we have to adopt a different database technology. We want to use both (and I'm sure many others do too)!

Suggested solution

We are using the .NET core version of Microsoft.Azure.DocumentDB.Core (version 1.9.1) and I can see that this uses System.Net.Http (version 4.1.0.0). The DocumentClient constructs a HttpClient, so the constructor could accept a HttpClientHandler and therefore the HttpClientHandler.ServerCertificateCustomValidationCallback could be used to override SSL certificate verification.

Could the ConnectionPolicy class be extended to provide a DisableSSLVerification property, to keep it similar with the Node SDK?

[kirankumarkolli]

.NET standard doesn’t have a reliable SSL validation callback (it has issues on non-windows environments and documented at https://github.com/dotnet/corefx/issues/19718 ). HTTP managed handler (SocketsHttpHandler) has the ability to address this gap and is still in development (Preview .NET standard 2.1 https://blogs.msdn.microsoft.com/dotnet/2018/02/27/announcing-net-core-2-1-preview-1/ check section “Sockets Performance and HTTP Managed Handler”).

Currently we are blocked on .NET standard to have this base capability for us to expose it.

[ElasticCoder]

Gotcha! I didn't realise that HttpClientHandler.ServerCertificateCustomValidationCallback had problems too! Then I guess we await .NET Core 2.1. Thanks!

[ElasticCoder]

Now that .NET Core 2.1 is out, do you have any updates on the roadmap for the client to allow the bypassing of SSL verification? I have written another .NET Core application that runs in a container and it wishes to connect to the emulator on the host, and cannot.

[Adam-MagmaSw]

Echoing the comment above from @ElasticCoder - this looks to be fairly simple now, do you guys have either an update on when you expect it to be done or an update on when the code will be made open source so we can do it?

[ealsur]

Support for a custom HttpMessageHandler was added in 2.2.0.

[dguinn-oncore]

Does anyone have an example or quick guidance as to how "Support for a custom HttpMessageHandler" solves this problem? It would be really nice for our development environments to be able to connect from a container to the host's cosmos db emulator. Thanks!

[ealsur]

@dguinn-oncore There is a new DocumentClient constructor that can receive an HttpMessageHandler. Basically if you look for SSL custom validation for NET Core, you will come up with samples like https://stackoverflow.com/a/44400067/5641598, https://stackoverflow.com/a/46626858/5641598. I believe an HttpClientHandler should also work.

[johngallardo]

Has anyone been successful in getting this approach to work when connecting to a CosmosDB emulator running in a Windows docker host for an aspnet core container? If I run curl https://host.docker.internal:8081 --insecure from within the container, that connects (i get an expected auth error, since i am not providing an authentication header, which is fine). However, connectivity via DocumentClient seems to just hang with the same using the same URI.

[despian]

I'm also having problems connecting to an emulator on a Windows VM on my Mac.

IntelliSense shows 8 overloaded versions of the DocumentClient constructor but none of them accepts an HttpClientHandler.

using Microsoft.Azure.Documents.Client; and version 2.2 of .NET Core

Edit. I was importing the wrong version of the package "Microsoft.Azure.DocumentDB.Core". Imported the newest version (2.2.1) and the additional constructor definition now shows up. Can confirm the method described in the StackOverflow post works.

[ElasticCoder]

It is now possible to run Linux containers using the dotnet core SDK and connect them to an emulator running on your development host. If anyone is interested, I have written a step-by-step guide here.

[dguinn-oncore]

@ElasticCoder - I'm glad you were finally able to get this to work, and thank you for the very detailed guide! Today, I updated our reference application to utilize the "V3 Beta Cosmos DB SDK" as you suggested and learned a lot about partitions as a result...

Anyways, I have not been able to get it to work following your guide. I'm hoping it's something really simple. When I start the emulator the second time, as you specify, using the following command:

"C:\Program Files\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe" /AllowNetworkAccess /NoFirewall /Key=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== /GetCert=cosmosdb.test

I get no response from the CLI (no thumbprint to copy), just an icon of the emulator starting in the windows taskbar.

Undeterred, I (think) I found the certificate in the "Personal/Certificates" area of the "Manage Windows Certificates" app you used. I exported it, following the instructions, then used your docker script to set up the cert from a new directory, which had the results you predicted.

After all that, I still get an "Invalid SSL Certificate" message when trying to connect to the emulator from my docker container. When I bin/bash into my container and try with curl, it gives me the "SSL Certificate Problem: Self Signed Certificate" error (same as it used to...). When I do it with the -k option, I get the correct response from the Cosmos dataexplorer.

I think this is occurring because the subject of the exported certificate is "localhost" and the various IP addresses of my host. It would need have a subject of "host.docker.internal" for it to work for me, I think... How are you referencing your host from your container?

Thanks for any help you can give me, and like I said, it's probably something I'm overlooking!

Thanks!

[ElasticCoder]

Hi @dguinn-oncore , I think something went wrong at the point you ran the emulator the second time and specified the /GenCert=cosmosdb.test command-line parameter. I've also spotted an error in my document, it is GenCert, not GetCert.

Here I would expect a dialog window to appear telling you that a new SSL certificate has been generated.

This certificate will now have cosmosdb.test as part of the subject hostnames and that is what I used to reference my emulator from a remote machine (or indeed from within the container). This is a comma-seperated list of hostnames so you could use /GetCert=cosmosdb.test,host.docker.internal if you wished (although I haven't tried that).

At that early part of the process I believe the /GetCert parameter was the only one I added, the others such as /AllowNetworkAccess and /Key come later once I have done the prep work. Maybe the other parameters change the startup behaviour and override the cert generation process? I'll check when I get a new machine.

I think something went wrong in that step for you and therefore you just found the original emulator certificate, the one which only supports localhost.

[ElasticCoder]

Well this is wierd. I've repeated it with the client and the emulator on the same machine and it doesn't work. Yesterday I had them on different machines. Let me investigate further.

[dguinn-oncore]

All -

I was able to get this to work with the following code based on the recommendation from @ealsur , which will work for us until we can get a certificate that works for both the emulator and container.

Thanks again for your help!

if (_documentDbOptions.BypassSSLCertificateValidation)
{
    /*
     * this allows for self-signed & invalid ssl certificates, which is currently required to connect from a docker 
     * container to the cosmos emulator...  
    */
    // todo:re-consider the need for this in a few months (02/12/19)
    var handler = new HttpClientHandler();
    handler.ClientCertificateOptions = ClientCertificateOption.Manual;
    handler.ServerCertificateCustomValidationCallback += (httpRequestMessage, x509Certificate2, x509Chain, sslPolicyErrors) =>
    {
        return true;
    };

    _client = new DocumentClient(new Uri(_documentDbOptions.Endpoint), _documentDbOptions.Key, handler);
}
else
{
    _client = new DocumentClient(new Uri(_documentDbOptions.Endpoint), _documentDbOptions.Key);
}

[ElasticCoder]

I've done a lot deeper digging today. There is a difference on SSL implementations between .NET core on Windows and Linux. (I don't know how I had it working last week, or thought I did).

On a Windows client (or Windows container), if you add the SSL certificate to the Trusted Root Certification Authorities node on the Certificate Store for the client computer, ,NET core will respect this and not complain about an self-signed certificate.

However when .NET core code runs under Linux it interops with the low-level OpenSSL libraries. When building the certificate chain it will identify a self-signed certificate and return an error code 18, which .NET core treats as a validation error This happens even if you add the self-signed certificate to the ca-certificates store on the image.

So you cannot configure the image to trust the emulator. You have to use the self-signed certificate bypass mechanism in code as you have demonstrated above. That is a slight shame as I would rather configre the image than alter the code between dev and production, but this is a limitation due to the interaction between .NET core and OpenSSL.

[parmindersinghkpt]

All -

I was able to get this to work with the following code based on the recommendation from @ealsur , which will work for us until we can get a certificate that works for both the emulator and container.

Thanks again for your help!

if (_documentDbOptions.BypassSSLCertificateValidation)
{
  /*
   * this allows for self-signed & invalid ssl certificates, which is currently required to connect from a docker 
   * container to the cosmos emulator...  
  */
  // todo:re-consider the need for this in a few months (02/12/19)
  var handler = new HttpClientHandler();
  handler.ClientCertificateOptions = ClientCertificateOption.Manual;
  handler.ServerCertificateCustomValidationCallback += (httpRequestMessage, x509Certificate2, x509Chain, sslPolicyErrors) =>
  {
      return true;
  };

  _client = new DocumentClient(new Uri(_documentDbOptions.Endpoint), _documentDbOptions.Key, handler);
}
else
{
  _client = new DocumentClient(new Uri(_documentDbOptions.Endpoint), _documentDbOptions.Key);
}

Hi dguinn-oncore

a) After bypass the ssl certificate validation i got the error (Connection refused) ---> System.Net.Http.HttpRequestException: Connection refused ---> System.Net.Sockets.SocketException: Connection refused.

b) I am not able to see HttpClientHandler and JsonSerializerSettings in single constructor.i need to set both HttpClientHandler and JsonSerializerSettings. I dont want to set JsonSerializerSettings in feed options.

I have turn off firewall

Would you please suggest me what to do ?

[SatyaKuppili]

Hello All, I have been facing an issue to start my local Azure Cosmos Emulator, Tried all the possible ways and reinstalled multiple times but still facing below issues in starting the emulator.

Any pointers would really helpful!!

Failed to acquire private key for Emulator secrets certificate. Error : 0x80090016

---

Azure Cosmos Emulator

Failed loading Emulator secrets certificate. Error: 0x8009000f