How to listen http and https with Kestrel and Service Fabric?

[odysseus1973]

Hi!

How to listen http and https with Kestrel and Service Fabric?

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(serviceContext =>
                    new KestrelCommunicationListener(serviceContext, "SSLServiceEndpoint", (url, listener) =>
                    {
                        ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                        return new WebHostBuilder()
                                    .UseKestrel(options =>
                                    {
                                        options.Listen(IPAddress.Any, 2000);
                                        options.Listen(IPAddress.Any, 3000, listenOptions =>
                                        {
                                            listenOptions.UseHttps(FindCertificate("cert_thumbprint", StoreLocation.LocalMachine,
                                                NameType.Thumbprint));
                                        });
                                    })
                                    .ConfigureServices(
                                        services => services
                                            .AddSingleton<StatelessServiceContext>(serviceContext))
                                    .UseContentRoot(Directory.GetCurrentDirectory())
                                    .UseStartup<Startup>()
                                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                    .UseUrls(url)
                                    .UseApplicationInsights()
                                    .Build();
                    }),"http")
            };

Port 3000 with https work correctly, but port 2000 with http not working.

[amanbha]

Does this work outside of SF in a standalone app?

[odysseus1973]

Don't try. How to test SF reliable service outside SF?

[amanbha]

Sorry if I wasn't clear enough. Not the stateful service, I only meant a standalone aspnet core app with the Kestrel options as your are setting above. This is to narrow down if it's a aspnet core Kestrel issue or SF issue.

[odysseus1973]

This is SF issue, on local machine asp net core app with Kestrel options work fine

[amanbha]

@odysseus1973 Ok, Could you share the service manifest. Whats the error for http@2000 you get? Also from your code snippet how do you plan to get the urls at which kestrel is listening from SF. SF will only returns the first url from IWebHost. If its stateless listening on static port and you know in advance then its probably fine. Usually the pattern with SF is to use one CommunicationListener per url, so that clients can get urls for the service replica by using SF resolve APIs.

[odysseus1973]

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="ApiCorePkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType. 
         This name must match the string used in RegisterServiceType call in Program.cs. -->
    <StatelessServiceType ServiceTypeName="ApiCoreType" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <!--<SetupEntryPoint>
      <ExeHost>
        <Program>scripts\launchConfig.cmd</Program>
      </ExeHost>
    </SetupEntryPoint>-->
    <EntryPoint>
      <ExeHost>
        <Program>ApiCore.exe</Program>
        <WorkingFolder>CodePackage</WorkingFolder>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
       independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
    <Endpoints>
      <!-- This endpoint is used by the communication listener to obtain the port on which to 
           listen. Please note that if your service is partitioned, this port is shared with 
           replicas of different partitions that are placed in your code. -->
      <Endpoint Protocol="http" Name="ServiceEndpoint" Type="Input" Port="2000" />
      <Endpoint Protocol="https" Name="SSLServiceEndpoint" Type="Input" Port="3000" />
    </Endpoints>
  </Resources>
</ServiceManifest>

This is stateless service with static port, which expose some api.

[amanbha]

Are you doing cert private key acl'ing your self? or its in app manifest. Please share appmanifest as well. Feel free to set up some time with me or send mail to amanbha@microsoft.com reg. this.

[odysseus1973]

SSL port 3000 working fine, http port 2000 not working, so this is not certificate related problem

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="ServiceFabric_ApiCoreType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="ApiCore_InstanceCount" DefaultValue="-1" />
  </Parameters>
  <!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion 
       should match the Name and Version attributes of the ServiceManifest element defined in the 
       ServiceManifest.xml file. -->
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="ApiCorePkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides />
    <Policies>
      <!--<RunAsPolicy CodePackageRef="Code" UserRef="SetupAdminUser" EntryPointType="Setup" />-->
      <!--<SecurityAccessPolicy ResourceRef="ServiceEndpoint" PrincipalRef="SetupAdminUser" />-->
      <!--<EndpointBindingPolicy EndpointRef="SSLServiceEndpoint" CertificateRef="asnacloud" />-->
    </Policies>
  </ServiceManifestImport>
  <DefaultServices>
    <!-- The section below creates instances of service types, when an instance of this 
         application type is created. You can also create one or more instances of service type using the 
         ServiceFabric PowerShell module.

         The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
    <Service Name="ApiCore" ServicePackageActivationMode="ExclusiveProcess">
      <StatelessService ServiceTypeName="ApiCoreType" InstanceCount="[ApiCore_InstanceCount]">
        <SingletonPartition />
        <PlacementConstraints>(name==D2v2)</PlacementConstraints>
      </StatelessService>
    </Service>
  </DefaultServices>
  <!--<Principals>
    <Users>
      <User Name="SetupAdminUser" AccountType="LocalSystem" />
    </Users>
  </Principals>-->
</ApplicationManifest>

netsh http show sslcert

IP:port                      : 0.0.0.0:3000
    Certificate Hash             : my_certificate_hash
    Application ID               : {1643619d-228d-4c4c-bcc9-8ad0f35bec7c}
    Certificate Store Name       : My
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled
    Reject Connections           : Disabled

netsh http show urlacl

    Reserved URL            : https://+:3000/
        User: NT AUTHORITY\NETWORK SERVICE
            Listen: Yes
            Delegate: No
            SDDL: D:(A;;GX;;;NS)

    Reserved URL            : https://10.0.0.4:3000/
        User: NT AUTHORITY\NETWORK SERVICE
            Listen: Yes
            Delegate: No
            SDDL: D:(A;;GX;;;NS)

    Reserved URL            : http://+:2000/
        User: NT AUTHORITY\NETWORK SERVICE
            Listen: Yes
            Delegate: No
            SDDL: D:(A;;GX;;;NS)

    Reserved URL            : http://10.0.0.4:2000/
        User: NT AUTHORITY\NETWORK SERVICE
            Listen: Yes
            Delegate: No
            SDDL: D:(A;;GX;;;NS)

[amanbha]

@odysseus1973 I am not able to repro this aspnetcore 2.0 and latest SF packages, i was able to access both http and https urls fine. Application17.zip

[odysseus1973]

@amanbha Thanks for answer. I try to publish you sample application with same result. Port 3000 https work perfect and port 2000 http not work. When I creating 2 ServiceInstanceListener, one for http and one for https, both ports works fine

[amanbha]

@odysseus1973 would it be possible for you to setup sometime so that I can take a look at your machine. Please send a meeting invite to amanbha@microsoft.com. Also could you provide following versions:

1. SF runtime version on your machine.
2. Microsoft.ServiceFabric.AspNetCore.Kestrel nuget package version.
3. Aspnet core nuget package version.

[odysseus1973]

Hi!

1. SF SDK version in my project 6.0.211 , in Azure SF version 6.0.211.9494
2. Microsoft.ServiceFabric.AspNetCore.Kestrel 2.8.211
3. AspNetCore 2.0.0

[vturecek]

@odysseus1973 Can you be more specific? What exactly do you mean by port 2000 "not working"? Does netstat show the application listening on port 2000? Do clients fail to resolve or connect?

[odysseus1973]

@vturecek

netstat -an
TCP    0.0.0.0:2000           0.0.0.0:0              LISTENING
TCP    10.0.0.4:2000          168.63.129.16:57059    ESTABLISHED

Output was cropped. Port 2000 accessed via localhost, but when I call from outside via browser I can not get an answer. At that time netstat -an show established connection from my IP.

@amanbha @vturecek Thanks for help

[amanbha]

@odysseus1973 This seems to be a machine specific issue as you are able to listen on other ports fine. I am closing the issue for now, feel free to reopen if more help is needed.

[brutaldev]

This is because you are only providing a single KestrelCommunicationListener and passing in the endpoint configuration name of "SSLServiceEndpoint". That endpoint uses port 3000 so you will only be able to connect through Service Fabric on that port and not on 2000. To use both HTTP and HTTPS (or multiple ports), you need to expose multiple KestrelCommunicationListener instances for each of your Service Fabric endpoint configurations.

Here is some dynamic code to do this for you by getting all the HTTP/HTTPS endpoints from the manifest and configuring Kestrel appropriately for each one based on the protocol.

var endpoints = Context.CodePackageActivationContext.GetEndpoints()
  .Where(endpoint => endpoint.Protocol == EndpointProtocol.Http || endpoint.Protocol == EndpointProtocol.Https);

return endpoints.Select(endpoint => new ServiceInstanceListener(serviceContext =>
  // New Service Fabric listener for each endpoint configuration in the manifest.
  new KestrelCommunicationListener(serviceContext, endpoint.Name, (url, listener) =>
  {
    return new WebHostBuilder()
      .UseKestrel(options =>
      {
        if (endpoint.Protocol == EndpointProtocol.Http)
        {
          options.Listen(IPAddress.Any, endpoint.Port);
        }
        else if (endpoint.Protocol == EndpointProtocol.Https)
        {
          options.Listen(IPAddress.Any, endpoint.Port, listenOptions =>  listenOptions.UseHttps(FindCertificate("cert_thumbprint", StoreLocation.LocalMachine,                                                NameType.Thumbprint)));
        }
      })
      .ConfigureServices(services => services.AddSingleton(serviceContext))
      .UseContentRoot(Directory.GetCurrentDirectory())
      .UseStartup<Startup>()
      .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
      .UseUrls(url)
      .Build();
  })));

[BuddhaBuddy1]

What NuGet package is FIndCertificate() and NameType defined? I'm using ASP.NET Core 2.0 and I don't see it anywhere.

[MorochoJaime]

Context.CodePackageActivationContext.GetEndpoints() HostingApplication.Context does not contain a definition for CodePackageActivationContext. Help please!!!

[Joostul]

@MorochoJaime context should be of type System.Fabric.StatelessServiceContext.

[nhdmalik]

Hi I am facing issue of infinite loop of calling CreateServiceInstanceListeners, I want to run on both http and https, can anyone see the attached application and guide me what is wrong?

TestHttpsApp.zip

[alexander-narbut]

Hi I am facing issue of infinite loop of calling CreateServiceInstanceListeners, I want to run on both http and https, can anyone see the attached application and guide me what is wrong?

TestHttpsApp.zip

@nhdmalik I had the same issue, fixed by passing "endpoint.Name" to the optional "name" parameter of ServiceInstanceListener

[Alex-Panov]

Hi I am facing issue of infinite loop of calling CreateServiceInstanceListeners, I want to run on both http and https, can anyone see the attached application and guide me what is wrong?

TestHttpsApp.zip

Had similar issue for ServiceReplicaListener in Stateful case, so it should look like this:

endpoints.Select(endpoint => new ServiceReplicaListener(serviceContext =>
                  // New Service Fabric listener for each endpoint configuration in the manifest.
                  new KestrelCommunicationListener(serviceContext, endpoint.Name, (url, listener) =>
                  {
...
                  }), endpoint.Name));

Note the second endpoint.Name on last line.