Need sample for using https endpoint with KestrelCommunicationListener

[blrchen]

I create a stateless asp.net core service and want to enable https endpoint for it. All samples I found are for WebListenerCommunicationListener. Since default asp.net core template in VS2017 is using KestrelCommunicationListener, can you add sample for KestrelCommunicationListener too?

[WhitWaldo]

I can't speak for the SF team, but here's the information you're looking for, mostly accumulated from https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?tabs=aspnetcore2x

Nuget Dependencies

Microsoft.ServiceFabric.AspNetCore.Kestrel Microsoft.AspNetCore.Authentication.Cookies Microsoft.AspNetCore.Authentication.OpenIdConnect

Your 'statelessservice'.cs file

/// <summary>
/// Optional override to create listeners (like tcp, http) for this service instance.
/// </summary>
/// <returns>The collection of listeners.</returns>
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
    ServiceEventSource.Current.ServiceMessage(Context, "CreateServiceInstanceListeners called.");

    return new ServiceInstanceListener[]
    {
        new ServiceInstanceListener(serviceContext =>
            new KestrelCommunicationListener(serviceContext, "EndpointHttps", (url, listener) =>
            {
                ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                return new WebHostBuilder()
                    .UseKestrel(opt =>
                    {
                        opt.Listen(IPAddress.Loopback, 443, listenOptions =>
                        {
                            listenOptions.UseHttps(GetCertificateFromStore());
                            listenOptions.NoDelay = true;
                        });
                    })
                    .ConfigureAppConfiguration((builderContext, config) =>
                    {
                        config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
                    })
                    .ConfigureServices(
                        services => services
                            .AddSingleton<StatelessServiceContext>(serviceContext))
                    .UseContentRoot(Directory.GetCurrentDirectory())
                    .UseStartup<Startup>()
                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                    .UseUrls(url)
                    .Build();
            }))
    };
}

private X509Certificate2 GetCertificateFromStore()
{
    var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
    try
    {
        store.Open(OpenFlags.ReadOnly);
        var certCollection = store.Certificates;
        var currentCerts = certCollection.Find(X509FindType.FindBySubjectDistinguishedName, "CN=<your_CN_value>", false);
        return currentCerts.Count == 0 ? null : currentCerts[0];
    }
    finally
    {
        store.Close();
    }
}

ServiceManifest.xml

Insert the following (in order to tell Service Fabric where to find the certificate): <Endpoint Protocol="https" Name="EndpointHttps" Type="Input" Port="443" CertificateRef="MySslCert"/>

ApplicationManifest.xml

In the <Parameters> at the top, add this: <Parameter Name="MySslCert" DefaultValue="" />

A little lower in the ServiceManifestImport for your service, create a new section under with:

    <Policies>
      <EndpointBindingPolicy EndpointRef="EndpointHttps" CertificateRef="HttpsCertificate" />
    </Policies>

And at the bottom, just before the </ApplicationManifest>, insert:

  <Certificates>
    <EndpointCertificate X509FindValue="[MySslCert]" Name="HttpsCertificate" />
  </Certificates>

Finally, in the appropriate ApplicationParameters (you might have different thumbprints for a prod and dev instance), within the <Parameters>: <Parameter Name="MySslCert" Value="<your thumbprint>" />

Good luck!

[masnider]

@rwike77 just to see if you think this should go in the howtos somewhere easily, or we can just close this since it's more about asp.net

[rfcdejong]

Nice example, I changed code from OwinCommunicationListener to KestrelCommunicationListener by converting to .ASP.NET NET Core. Only missing code for statefull services that was in the custom OwinCommunicationListener.

Still getting the following exception with multiple endpoints in the code above:

"Unique Name must be specified for each listener when multiple communication listeners are used"

[rwike77]

@blrchen @masnider Yes, I think we do need an example of this in our docs. I'm working on a tutorial segment right now. @WhitWaldo, good example.

[barelabs]

A tutorial would really help a lot. I'm having trouble from the default template package that Visual Studio installs automatically. When an MVC/webapi project is created without service fabric, visual studio automatically prompts you to install a certificate to make local development work with SSL work, but when making a new project with Visual Studio using the service fabric templates for MVC/api (with built in authentication), nothing loads, even on first run of application. It just says site can't be reached...why doesn't the default template (when creating a new MVC project with user authentication with a Service Fabric new project) automatically set up SSL configuration like the regular default templates in visual studio do?

[GopiMtp]

@rwike77 yeah, Sample to host web api's thru HTTPS would be a definite need in MS Docs. It would definately help lot of SF beginners like me.

[rwike77]

@GopiMtp @firecapella @blrchen Apologies for the delay on this. The process is actually different than what @WhitWaldo outlined previously. I'm working out some details, hopefully this will get you unblocked for now.

Using Kestrel, you don't need to update the ApplicationManifest.xml file at all. The ServiceManifest.xml file needs to define the HTTPS endpoint.

You need to import the certificate into the LocalMachine\My store. The service, running under NETWORK SERVICE by default, needs access to the cert's private key. To manually do this on your dev box, open up certlm.msc, expand Personal->Certificates, and right-click your cert. Select All Tasks->Manage private keys and then add NETWORK SERVICE.

That should get you debugging locally. You'll need to install the cert on all the nodes of the remote cluster. First, upload your cert to a keyvault and then run Add-AzureRmServiceFabricApplicationCertificate to install it on all the cluster nodes. On each of the cluster nodes, you still need to give NETWORK SERVICE access to the private key. I'm still working out final details, but you should be able to run some PowerShell in the service SetupEntryPoint of the service to do this. Here are some specifics, again sorry for the delay. Hope this helps.

Your WebService.cs file

This example uses port 443 the EndpointHttps endpoint defined in the service manifest.

/// <summary>
        /// Optional override to create listeners (like tcp, http) for this service instance.
        /// </summary>
        /// <returns>The collection of listeners.</returns>
        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(
    serviceContext =>
        new KestrelCommunicationListener(
            serviceContext,
            "EndpointHttps",
            (url, listener) =>
            {
                ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                return new WebHostBuilder()
                    .UseKestrel(opt =>
                    {
                        opt.Listen(IPAddress.IPv6Any, 443, listenOptions =>
                        {
                            listenOptions.UseHttps(GetCertificateFromStore());
                            listenOptions.NoDelay = true;
                        });
                    })
                    .ConfigureAppConfiguration((builderContext, config) =>
                    {
                        config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
                    })

                    .ConfigureServices(
                        services => services
                            .AddSingleton<HttpClient>(new HttpClient())
                            .AddSingleton<FabricClient>(new FabricClient())
                            .AddSingleton<StatelessServiceContext>(serviceContext))
                    .UseContentRoot(Directory.GetCurrentDirectory())
                    .UseStartup<Startup>()
                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                    .UseUrls(url)
                    .Build();
            }))
            };
        }

        private X509Certificate2 GetCertificateFromStore()
        {
            var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
            try
            {
                store.Open(OpenFlags.ReadOnly);
                var certCollection = store.Certificates;
                var currentCerts = certCollection.Find(X509FindType.FindBySubjectDistinguishedName, "CN=localhost", false);
                return currentCerts.Count == 0 ? null : currentCerts[0];
            }
            finally
            {
                store.Close();
            }
        }

ServiceManifest.xml

Create the HTTPS endpoint in Endpoints:

<Resources>
    <Endpoints>
      <!-- This endpoint is used by the communication listener to obtain the port on which to 
           listen. Please note that if your service is partitioned, this port is shared with 
           replicas of different partitions that are placed in your code. -->
      <Endpoint Protocol="https" Name="EndpointHttps" Type="Input" Port="443" />
    </Endpoints>
  </Resources>

Install the cert on the cluster nodes

Upload the cert to keyvault and install on the cluster nodes:

Connect-AzureRmAccount

$vaultname="sftestvault"
$certname="VotingApp2PFX"
$certpw="!Password321#"
$groupname="voting_RG"
$clustername = "votinghttps"
$ExistingPfxFilePath="C:\Users\sfuser\votingappcert.pfx"

$appcertpwd = ConvertTo-SecureString –String $certpw –AsPlainText –Force  

Write-Host "Reading pfx file from $ExistingPfxFilePath"
$cert = new-object System.Security.Cryptography.X509Certificates.X509Certificate2 $ExistingPfxFilePath, $certpw

$bytes = [System.IO.File]::ReadAllBytes($ExistingPfxFilePath)
$base64 = [System.Convert]::ToBase64String($bytes)

$jsonBlob = @{
   data = $base64
   dataType = 'pfx'
   password = $certpw
   } | ConvertTo-Json

$contentbytes = [System.Text.Encoding]::UTF8.GetBytes($jsonBlob)
$content = [System.Convert]::ToBase64String($contentbytes)

$secretValue = ConvertTo-SecureString -String $content -AsPlainText -Force

# Upload the certificate to the key vault as a secret
Write-Host "Writing secret to $certname in vault $vaultname"
$secret = Set-AzureKeyVaultSecret -VaultName $vaultname -Name $certname -SecretValue $secretValue

# Add a certificate to all the VMs in the cluster.
Add-AzureRmServiceFabricApplicationCertificate -ResourceGroupName $groupname -Name $clustername -SecretIdentifier $secret.Id -Verbose

On cluster nodes, give NETWORK SERVICE access to private key

This is the part I'm still working out. You should be able to run something like the following in the service SetupEntryPoint. Assuming the cert is already on the cluster node, the service setup will run the script before the service code (and Kestrel) runs.

$subject="localhost"
$userGroup="NETWORK SERVICE"

Write-Host "Checking permissions to certificate $subject.." -ForegroundColor DarkCyan

$cert = (gci Cert:\LocalMachine\My\ | where { $_.Subject.Contains($subject) })[-1]

if ($cert -eq $null)
{
    $message="Certificate with subject:"+$subject+" does not exist at Cert:\LocalMachine\My\"
    Write-Host $message -ForegroundColor Red
    exit 1;
}elseif($cert.HasPrivateKey -eq $false){
    $message="Certificate with subject:"+$subject+" does not have a private key"
    Write-Host $message -ForegroundColor Red
    exit 1;
}else
{
    $certHash = $cert.Thumbprint
    $certSubject = $cert.Subject
    $cert.PrivateKey
    $keyName=$cert.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName

    $keyPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\"
    $fullPath=$keyPath+$keyName
    $acl=(Get-Item $fullPath).GetAccessControl('Access')

    $hasPermissionsAlready = ($acl.Access | where {$_.IdentityReference.Value.Contains($userGroup.ToUpperInvariant()) -and $_.FileSystemRights -eq [System.Security.AccessControl.FileSystemRights]::FullControl}).Count -eq 1

    if ($hasPermissionsAlready){
        Write-Host "Account $userGroupCertificate already has permissions to certificate '$subject'." -ForegroundColor Green
        return $true;
    } else {
        Write-Host "Need add permissions to '$subject' certificate..." -ForegroundColor DarkYellow

        $permission=$userGroup,"Full","Allow"
        $accessRule=new-object System.Security.AccessControl.FileSystemAccessRule $permission
        $acl.AddAccessRule($accessRule)
        Set-Acl $fullPath $acl

        Write-Output "Permissions were added"

        return $false;
    }
}

[GopiMtp]

@rwike77 yes, this definitely helped me to setup https on my local machine. The Cert name is kind of hard coded and works for "Localhost" certs, but if i need to work with cloud, this should be configurable based on my environment. Do you have insights on how to do that? I'm trying to find that option currenlty.

[rwike77]

Posted a tutorial for enabling HTTPS in an ASP.NET Core web service.

[rwike77]

@GopiMtp Yes, the cert name is hard coded right now. I'll make updates when I can. Take a look at using the appsettings.json file for using different configuration settings in different environments.

[rwike77]

Closing this issue, since I posted the tutorial.

[Cular]

@rwike77 , thank you for great tutorial. I have only one question, probably call Set-Acl $fullPath $acl from PS (in Add the batch and PowerShell setup scripts) should be wrapped on try/catch, on debugging it will simplify understanding where the problem. My case was in localizated SystemGroup, for example.

Try
{
    Set-Acl $fullPath $acl -ErrorAction Stop
    Write-Output "Permissions were added"
}
Catch
{
    Write-Host "Error:"$_.Exception.Message -ForegroundColor Red
    exit 1;
}

[payiAzure]

@rwike77 , saw your post in the issue and currently having an issue while following the tutorial you posted.

1. "install cert" step in the tutorial. The cert installed and used by ssl should be the "cluster cert" or "reverse proxy ssl cert" of the service fabric cluster?

2. I have a asp.net core web api service running on service fabric cluster. Originally, I use http endpoint, and call the api from web browser like: http://*****.westus2.cloudapp.azure.com:80/api/values The call went through, return data as expected. Port 80 is the loadbalencer frontend port.

   Then I want to use https endpoint. I did the setup following tutorial. But when I call web api with the following, I got ERR_HTTP2_INADEQUATE_TRANSPORT_SECURITY error, the following endpoints couldn't be readched: https://*****.westus2.cloudapp.azure.com:443/api/values

   I did a lot of search, but couldn't figure out where I did wrong. Could you point me where might cause the issue?

Thanks

[rwike77]

I haven't worked on Service Fabric for a while, @erikadoyle can you take a look at this? Thanks.

[rwike77]

reassign:@erikadoyle

[rwike77]

assign:@erikadoyle

[rwike77]

reassign @erikadoyle