Gateway can't expose two endpoints on different ports

[markheath]

I've created a JSON application template and want to use a gateway to expose two endpoints publicly. I have two services (voteService and resultService) each with an endpoint (VoteListener and ResultListener) and want to expose them both on different ports. I can't simply use two routes (as shown in the "ingress" sample) for the same port as my services don't work correctly with a route prefix.

I've tried various combinations of http and tcp gateway configurations, but every time I deploy only one of the two endpoints is accessible (sometimes the vote one, sometimes the result one).

My most recent attempt is this:

        "http": [
          {
            "name": "vote",
            "port": 80,
            "hosts": [
              {
                "name": "*",
                "routes": [
                  {
                    "name":  "voteRoute",
                    "match": {
                      "path": {
                        "value": "/",
                        "rewrite": "/",
                        "type": "Prefix"
                      }
                    },
                    "destination": {
                      "applicationName": "votingApp",
                      "serviceName": "voteService",
                      "endpointName": "VoteListener"
                    }
                  }
                ]
              }
            ]
          },
          {
            "name": "result",
            "port": 8082,
            "hosts": [
              {
                "name": "*",
                "routes": [
                  {
                    "name": "resultRoute",
                    "match": {
                      "path": {
                        "value": "/",
                        "rewrite": "/",
                        "type": "Prefix"
                      }
                    },
                    "destination": {
                      "applicationName": "votingApp",
                      "serviceName": "resultService",
                      "endpointName": "ResultListener"
                    }
                  }
                ]
              }
            ]
          }
        ],
        "tcp": [
          {
            "name": "tcpVote",
            "port": 8080,
            "destination": {
              "applicationName": "votingApp",
              "serviceName": "voteService",
              "endpointName": "VoteListener"
            }
          },
          {
            "name": "tcpResult",
            "port": 8081,
            "destination": {
              "applicationName": "votingApp",
              "serviceName": "resultService",
              "endpointName": "ResultListener"
            }
          }
        ]

And this exposed the Result service on both 8081 and 8082 but the vote service was not accessible on 80 or 8080.

The containers are windows if that makes any difference. From examining the container logs, all containers seem to have started successfully.

I also tried defining two different gateways each exposing one service but that also didn't work.

Is this a limitation of the Service Fabric Mesh preview, or am I doing something fundamentally wrong?

[guibirow]

For security reasons I didn't run your container images, I've modified the HelloWorld sample and added and extra service and the gateway added multiple routes (1 http for each service and 1 tcp for each), I can confirm All routes worked "without issues":

This is my gateway configuration:

    {
      "apiVersion": "2018-09-01-preview",
      "name": "helloWorldGateway",
      "type": "Microsoft.ServiceFabricMesh/gateways",
      "location": "[parameters('location')]",
      "dependsOn": [
        "Microsoft.ServiceFabricMesh/networks/helloWorldNetwork"
      ],
      "properties": {
        "description": "Service Fabric Mesh Gateway for HelloWorld sample.",
        "sourceNetwork": {
          "name": "Open"
        },
        "destinationNetwork": {
          "name": "[resourceId('Microsoft.ServiceFabricMesh/networks', 'helloWorldNetwork')]"
        },
        "http": [
          {
            "name": "web",
            "port": 81,
            "hosts": [
              {
                "name": "*",
                "routes": [
                  {
                    "name":  "helloRoute",
                    "match": {
                      "path": {
                        "value": "/",
                        "rewrite": "/",
                        "type": "Prefix"
                      }
                    },
                    "destination": {
                      "applicationName": "helloWorldApp",
                      "serviceName": "helloWorldService",
                      "endpointName": "helloWorldListener"
                    }
                  }
                ]
              }
            ]
          },
          {
            "name": "kuard",
            "port": 82,
            "hosts": [
              {
                "name": "*",
                "routes": [
                  {
                    "name":  "kuardRoute",
                    "match": {
                      "path": {
                        "value": "/",
                        "rewrite": "/",
                        "type": "Prefix"
                      }
                    },
                    "destination": {
                      "applicationName": "helloWorldApp",
                      "serviceName": "kuardService",
                      "endpointName": "kuardListener"
                    }
                  }
                ]
              }
            ]
          }
        ],
        "tcp": [
          {
            "name": "web",
            "port": 80,
            "destination": {
              "applicationName": "helloWorldApp",
              "serviceName": "helloWorldService",
              "endpointName": "helloWorldListener"
            }
          },
          {
            "name": "kuard",
            "port": 8080,
            "destination": {
              "applicationName": "helloWorldApp",
              "serviceName": "kuardService",
              "endpointName": "kuardListener"
            }
          }
        ]
      }
    }

When I hit the endpoints:

http://gatewayip:80 and http://gatewayip:81 routes to helloWorldService

http://gatewayip:82 and http://gatewayip:8080 routes to kuardService

PS: At the top, I gave an emphasis on "without issues", because seems that the gateway does not work while being upgraded, this issue is being tracked on https://github.com/Azure/service-fabric-mesh-preview/issues/304

[markheath]

Thanks, good to know it ought to work although I still can't understand why mine isn't working. Could it be to do with using Windows containers? The strange thing is that each time I deploy it I seem to get different results. Sometimes neither service is accessible, sometimes it is the first (vote) service, other times the second (result) is available.

On my last attempt, the vote service was available on both (http and tcp) configrued ports, but the result service was not - I get "upstream connect error or disconnect/reset before headers"

I'll keep experimenting, and maybe try to reproduce your successful example

[guibirow]

it is possible, I didn't have the chance to test it with Windows containers yet.

I would recommend you first deploy one service and one route and test if the services are working correctly, then add another service and then add other routing

I've noticed a strange behavior on Gateway when we apply updates the gateway stop working, not sure why, might be related to the #304 reported above.

[markheath]

By the way, sometimes when I get my app status using az mesh app show, I see the following error: There was an error during activation.IPAddressProviderAddressRangeExhausted

[guibirow]

Not sure about this issue, make sure you are not hitting the max quotas described here

[sridmad]

@markheath, Can you open a different issue for IPAddressProviderAddressRangeExhausted error?

Windows vs Linux containers: Depending on the base image used for Windows image, it can take a while to download the image and hence not reachable for a while. Unless you need Windows Server Core base image, use one of the nanoserver images as base image. Some times I have seen images based on Windows Server core take upwards of 5 minutes to download.

We will update the Hello World conatiner to use nano server to speed up download and activation.

I get "upstream connect error or disconnect/reset before headers"

Does it work after a while. This most probably means the container is not yet activated completely.

[aloneguid]

I've just tested and it works for me, two services on different ports are accessible. My gateway resource:

gateway:
  schemaVersion: 1.0.0-preview2
  name: TwoApisGateway
  properties:
    description: TwoApis Gateway Resource
    sourceNetwork:
      name: Open
    destinationNetwork:
      name: TwoApisNetwork
#    tcp:
#      - name: FactsWebSiteTCP
#        port: "[parameters('FactsWebSite_ingressport')]"
#        destination:
#          applicationName: TwoApis
#          serviceName: FactsWebSite
#          endpointName: FactsWebSiteListener
#      - name: FactsApiTCP
#        port: "[parameters('FactsApi_ingressport')]"
#        destination:
#          applicationName: TwoApis
#          serviceName: FactsApi
#          endpointName: FactsApiListener
    http:
      - name: FactsApiHTTP
        port: 80
        hosts:
          - name: "*"
            routes:
              - name: api
                match:
                  path:
                    value: "/api/"
                    rewrite: "/"
                    type: Prefix
                destination:
                  applicationName: TwoApis
                  serviceName: FactsApi
                  endpointName: FactsApiListener
      - name: FactsWebHTTP
        port: 81
        hosts:
          - name: "*"
            routes:
              - name: web
                match:
                  path:
                    value: "/"
                    rewrite: "/"
                    type: Prefix
                destination:
                  applicationName: TwoApis
                  serviceName: FactsWebSite
                  endpointName: FactsWebSiteListener     

[markheath]

OK thanks, I guess there must be something weird about my containers. I did use to have it working exposing two endpoints with the previous JSON schema, but something broke a week ago and haven't managed to get it going again with either the 2018-07-01-preview or 2018-09-01-preview api version despite trying out dozens of different configurations.

[markheath]

@sridmad yes I'll open another issue if I see it again. It seems to be fairly common in this application although the last few deployments didn't have this error.

I've tried waiting but the problem doesn't seem to resolve itself.

I'm still testing various combinations. Sometimes neither service is available.

[markheath]

OK, finally after numerous deployments, I've seen it work! Didn't change my ARM template at all, just deleted and redeployed. Here's the template I was using.

[markheath]

I've been doing daily deployments of the same template, and unfortunately it has never worked since. Of the two services I expose, usually the voteListener endpoint is accessible and the resultListener one isn't. Today it was the other way round. It doesn't seem able to recover from this situation despite waiting several hours.

[julipur]

I have been fighting with the same issue and have tried various different ways including adding http as well as shown in some examples. I have 4 services and I can only get two to work (and that too randomly). I'm using the nano server image as well.