Question: On Spring Cloud Ribbons vs Go Router

amarflybot commented 7 years ago

This is question: Please drop links if you think it is answered before. Since "Go Router" is also a routing tool, does it mean we shall not have "Discovery Server" in our infrastructure. Spring Cloud Eureka does exactly what this router provides. Also is this router smart enough to know that the underneath Spring API Service is avaliable to get consumed?

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/134901087

The labels on this github issue will be updated when the story is started.

shashwathi commented 7 years ago

@amarraja123 :

Does it mean we shall not have "Discovery Server" in our infrastructure. Spring Could Eureka does exactly what this router provides.

The GoRouter is, in simple terms, a reverse proxy that load balances between many backend instances. According to my knowledge "Discovery Server" receives backend registration through a client and gorouter does not function this way. I am not aware of the project Spring Could Eureka so I have no idea whether it is similar or not. Examples of Discovery server: etcd, consul.

Also is this router smart enough to know that the underneath Spring API Service is available to get consumed?

Gorouter does not discriminate between any backends. It does not know if backend is Spring app or not.

If you do not have any further questions, feel free to close this issue.

Regards Shash

amarflybot commented 7 years ago

@shashwathi Thanks for replying. Could you please share some docs or metrics to campare the difference and requests between applications using Discovery Server vs without discovery server. I understand that one is client reporting the server vs other is the server pinging its child.

aaronshurley commented 7 years ago

@amarraja123 :

Could you please share some docs or metrics to campare the difference and requests between applications using Discovery Server vs without discovery server.

There is no comparison doc or metrics between these components.

understand that one is client reporting the server vs other is the server pinging its child.

In gorouter, we rely on route-registrar and route-emitter to register the backends. All the registration msgs go through NATS and all gorouters are subscribed to these msgs. There is a good amount documentation on README about internals of gorouter.

Regards @shashwathi && Aaron

shashwathi commented 7 years ago

Hi @amarraja123

Closing this issue because of inactivity. Feel free to reopen this issue or create a new one for further questions.

Regards Shash

shalako commented 7 years ago

@amarraja123 To draw parallels with a service discovery system, services are registered with gorouter via NATS if the service receives http requests, and with routing api if the service receives requests over other tcp protocols. Service discovery is different however. Clients of a service must know the route of the service (such as some-service.apps.cf.com) and when DNS resolves this hostname to a load balancer in front of a tier of gorouters, and forwards the request to the gorouters, then the routers forward the request to the registered service.

amarflybot commented 7 years ago

Thanks for all the clarification !!

ramaobilisetty commented 7 years ago

Thanks Shalako. Can you please elaborate on each step you mentioned? We have been analyzing on Eureka Vs GoRouter and at some point we thought that Eureka may not be required at all but after seeing your comments, it seems both are required.

charlesroopesh commented 7 years ago

Hi - We are in a similar situation where we are using PCF. Now I believe we are looking at two patterns i.e. Service Discovery at server side (is this what Go Router is ?) and client side (Eureka). Considering all our microservices sit within PCF - do we need both ? Or is Go Router not a service discovery component and merely a reverse proxy ? And also in https://docs.pivotal.io/spring-cloud-services/1-3/common/service-registry/ it states "Service Registry for Pivotal Cloud Foundry is based on Eureka" - so what does this mean with Go Router into play ?

shashwathi commented 7 years ago

Reopening this issue to address concerns. @shalako has more context on the issue above.

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/146799991

The labels on this github issue will be updated when the story is started.

shalako commented 7 years ago

@ramaobilisetty @charlesroopesh

Gorouter is the edge reverse proxy for Cloud Foundry. It manages inbound http requests and their responses for apps as well as the CF platform control plane.

Example data paths for requests from external clients: web-browser [DNS for appA hostname] --> load balancer --> Gorouter --> appA cf CLI [DNS for CF Cloud Controller API (CAPI)] --> load balancer --> Gorouter --> CAPI

Outbound requests from apps on CF to external servers do not transit Gorouter. It is common for operators to maintain a NAT component through which outbound connections are proxied so that the VMs running app containers do not require interfaces on a public network.

appA [DNS] --> NAT --> external server

Gorouter leverages platform service discovery mechanisms that are purpose built for Gorouter. Platform system components register the location (IP and port) of every app instance as well as the platform control APIs that are publicly routable, along with their routable addresses through an internal message bus called NATS, to which the Gorouter is subscribed. In this way, Gorouter maintains the mapping between routable addresses (URIs) and all routable endpoints (IP and ports) in the platform.

For traffic between applications running on CF, the default data path would be for appA to use the publicly routable address for appB:

appA [DNS for appB hostname] --> load balancer --> Gorouter --> appB

You may choose to operator your own service discovery mechanism (e.g. Eureka or Pivotal Spring Cloud Services) to map service names to these routable addresses. The data path still goes through the platform front door (edge LB + gorouter).

You may choose to use private load balancers and private DNS to route inter-app requests through a private front door, but the requests still go through Gorouter, as it is the proxy that tracks the dynamic location of all routable endpoints.

You may choose to use Diego Container Networking (aka container-to-container networking) to enable appA to contact appB directly, with no intermediating components, via an overlay network. In this scenario, requests do not go through the edge LB and Gorouter. However, there is no built-in service discovery mechanism for this strategy so you must provide your own. You might use Eureka or Pivotal Spring Cloud Services to manage the registration and discovery of the mapping of service names to IPs and ports. Each application leveraging this strategy would need to be pushed with a client on each app instance that discovers the current IP and PORT and registers it with the service you operate. Your apps that consume these services must also subscribe to the service discovery system you operate.

appA [your service discovery] --> appB

As I hope the above illustrates, we expect you may use both Gorouter and some service discovery system (e.g. Eureka); Gorouter for requests from clients external to the platform, and Eureka for requests between apps running on the platform.

shubhaat commented 6 years ago

Closing this issue for now since it's been open for a year with no activity. We have since launched service discovery for C2C communication using BOSH DNS documented here: https://docs.cloudfoundry.org/devguide/deploy-apps/cf-networking.html#discovery for the use case appA [BOSH DNS for service discovery] --> appB. For details on this you could ask the cf-networking-release team here: https://github.com/cloudfoundry/cf-networking-release.

Let us know if we can help more.

TheFonz2017 commented 5 years ago

Nice discussion! Thanks a lot. This helped to better understand Go Router and CF routing.

Having worked with Ribbon and Eureka lately, I was thinking of the following - which should be possible and might even be a candidate for integration with Spring Cloud:

The Facts

Cloud Foundry has the possibility not only to enable container-to-container communication but also to set a request header with the application instance ID that Go-Router should forward the request to. Basically, this boils down to adding an HTTP header of the form "X-CF-APP-INSTANCE":"YOUR-APP-GUID:YOUR-INSTANCE-INDEX"- notice that the App GUID is required and the instance index This is described here. Usually this is used for debugging.
Ribbon (when used with Eureka) gets its server / service instance list from Eureka and will balance load between them (e.g. in Round Robin fashion or based on a custom rule).

In a common Cloud Foundry deployment, all instances of an application will share the same (external) route. Thus Ribbon might see 3 different instances, but since they all have the same route, it cannot effectively distinguish the actual instance a request to that route will be forwarded to. That's in the hands of Go-Router.

The Idea

If Ribbon could derive an application instance index and get hold of the Application GUID, it could add above HTTP header to explicitly tell Go-Router where to route its request to.

A Possible Solution

Ribbon has access to the InstanceInfo of each service instance in the server list returned by Eureka (see the appendix below). In particular Ribbon knows how many instances of the application there are - allowing it to derive instance indices (0 to #nInstances-1). InstanceInfo also contains (among many other things) the instanceIds (as of vcap.application.instance_id) of the application instances, as well as their metadata - which you can add custom fields to.

InstanceInfo does not contain the application GUID which is required, but we could add this as custom metadata in your application's application.yml like so:

eureka:
  instance:
    metadata-map:
      applicationId: ${vcap.application.application_id} # see: https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-APPLICATION

We could then have a Ribbon rule, that based on the number of application instances and the applicationGUID from metadata

determines instance indices
adds the "X-CF-APP-INSTANCE":"AppGUID:InstanceIndex" to HTTP requests

Effectively this would tell Go-Router exactly where to route.
Of course this is not an ideal solution - actually Go-Router is then just an unnecessary extra hop. However, for simple scenarios, this might be simple solution.

Appendix

Here is what the InstanceInfo returned by Eureka looks like for an application with 3 instances. Notice the applicationId in the metadata node.

[
  {
    "host": "address-service.my.domain.com",
    "port": 443,
    "metadata": {
      "management.port": "8080",
      "cluster": "DEV",
      "version": "1.0.0",
      "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
    },
    "uri": "https://address-service.my.domain.com:443",
    "instanceId": "f4bc4d54-40c2-459d-6241-b70e",
    "serviceId": "ADDRESS-SERVICE",
    "secure": true,
    "instanceInfo": {
      "instanceId": "f4bc4d54-40c2-459d-6241-b70e",
      "app": "ADDRESS-SERVICE",
      "appGroupName": null,
      "ipAddr": "<some IP address>",
      "sid": "na",
      "homePageUrl": "http://address-service.my.domain.com:8080/",
      "statusPageUrl": "http://address-service.my.domain.com:8080/actuator/info",
      "healthCheckUrl": "http://address-service.my.domain.com:8080/actuator/health",
      "secureHealthCheckUrl": "address-service.my.domain.com:8080/actuator/health",
      "vipAddress": "address-service",
      "secureVipAddress": "address-service",
      "countryId": 1,
      "dataCenterInfo": {
        "@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo",
        "name": "MyOwn"
      },
      "hostName": "address-service.my.domain.com",
      "status": "UP",
      "overriddenStatus": "UNKNOWN",
      "leaseInfo": {
        "renewalIntervalInSecs": 30,
        "durationInSecs": 90,
        "registrationTimestamp": 1552482528374,
        "lastRenewalTimestamp": 1552483821431,
        "evictionTimestamp": 0,
        "serviceUpTimestamp": 1552482528374
      },
      "isCoordinatingDiscoveryServer": false,
      "metadata": {
        "management.port": "8080",
        "cluster": "DEV",
        "version": "1.0.0",
        "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
      },
      "lastUpdatedTimestamp": 1552482528374,
      "lastDirtyTimestamp": 1552482528207,
      "actionType": "ADDED",
      "asgName": null
    },
    "scheme": null
  },
  {
    "host": "address-service.my.domain.com",
    "port": 443,
    "metadata": {
      "management.port": "8080",
      "cluster": "DEV",
      "version": "1.0.0",
      "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
    },
    "uri": "https://address-service.my.domain.com:443",
    "instanceId": "3676f5a0-7a2d-4bda-43e2-3bf2",
    "serviceId": "ADDRESS-SERVICE",
    "secure": true,
    "instanceInfo": {
      "instanceId": "3676f5a0-7a2d-4bda-43e2-3bf2",
      "app": "ADDRESS-SERVICE",
      "appGroupName": null,
      "ipAddr": "<some IP address>",
      "sid": "na",
      "homePageUrl": "http://address-service.my.domain.com:8080/",
      "statusPageUrl": "http://address-service.my.domain.com:8080/actuator/info",
      "healthCheckUrl": "http://address-service.my.domain.com:8080/actuator/health",
      "secureHealthCheckUrl": "https://address-service.my.domain.com:8080/actuator/health",
      "vipAddress": "address-service",
      "secureVipAddress": "address-service",
      "countryId": 1,
      "dataCenterInfo": {
        "@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo",
        "name": "MyOwn"
      },
      "hostName": "address-service.my.domain.com",
      "status": "UP",
      "overriddenStatus": "UNKNOWN",
      "leaseInfo": {
        "renewalIntervalInSecs": 30,
        "durationInSecs": 90,
        "registrationTimestamp": 1552482537928,
        "lastRenewalTimestamp": 1552483802487,
        "evictionTimestamp": 0,
        "serviceUpTimestamp": 1552482537928
      },
      "isCoordinatingDiscoveryServer": false,
      "metadata": {
        "management.port": "8080",
        "cluster": "DEV",
        "version": "1.0.0",
        "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
      },
      "lastUpdatedTimestamp": 1552482537928,
      "lastDirtyTimestamp": 1552482537616,
      "actionType": "ADDED",
      "asgName": null
    },
    "scheme": null
  },
  {
    "host": "address-service.my.domain.com",
    "port": 443,
    "metadata": {
      "management.port": "8080",
      "cluster": "DEV",
      "version": "1.0.0",
      "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
    },
    "uri": "https://address-service.my.domain.com:443",
    "instanceId": "a4392a32-e137-4d76-5155-7b2c",
    "serviceId": "ADDRESS-SERVICE",
    "secure": true,
    "instanceInfo": {
      "instanceId": "a4392a32-e137-4d76-5155-7b2c",
      "app": "ADDRESS-SERVICE",
      "appGroupName": null,
      "ipAddr": "<some IP address>",
      "sid": "na",
      "homePageUrl": "http://address-service.my.domain.com:8080/",
      "statusPageUrl": "http://address-service.my.domain.com:8080/actuator/info",
      "healthCheckUrl": "http://address-service.my.domain.com:8080/actuator/health",
      "secureHealthCheckUrl": "https://address-service.my.domain.com:8080/actuator/health",
      "vipAddress": "address-service",
      "secureVipAddress": "address-service",
      "countryId": 1,
      "dataCenterInfo": {
        "@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo",
        "name": "MyOwn"
      },
      "hostName": "address-service.my.domain.com",
      "status": "UP",
      "overriddenStatus": "UNKNOWN",
      "leaseInfo": {
        "renewalIntervalInSecs": 30,
        "durationInSecs": 90,
        "registrationTimestamp": 1552481701564,
        "lastRenewalTimestamp": 1552483805329,
        "evictionTimestamp": 0,
        "serviceUpTimestamp": 1552481701564
      },
      "isCoordinatingDiscoveryServer": false,
      "metadata": {
        "management.port": "8080",
        "cluster": "DEV",
        "version": "1.0.0",
        "applicationId" : "3cfa403a-c01c-42bd-978a-fcd296cfb0d7"
      },
      "lastUpdatedTimestamp": 1552481701564,
      "lastDirtyTimestamp": 1552481701556,
      "actionType": "ADDED",
      "asgName": null
    },
    "scheme": null
  }
]

TheFonz2017 commented 5 years ago

I have implemented above-mentioned approach. If anyone wants to have a look at it, you can find it here: https://github.com/TheFonz2017/Spring-Cloud-Netflix-Ribbon-CF-Routing

Cheers.

cloudfoundry / gorouter