How does discovery work with container networking?

[dsyer]

Maybe something magic happens and it all works just fine. Would be good to try it and document it at least.

[spencergibb]

@csterwa, @royclarkson or @scottfrederick could give us a little insight.

[scottfrederick]

Discovery "just works" with container networking. Some configuration is required in the apps and in the platform if you want the registering app and discovering app to communicate directly via IP address (i.e. container to container, bypassing the platform router). There's a blog post that provides some details, with Spring Cloud Services specifics.

Setting spring.cloud.services.registrationMethod=direct as described in the blog post is just shorthand for setting eureka.instance.preferIpAddress=true and eureka.instance.nonSecurePort=${PORT}.

[scottfrederick]

Revisiting this now that I have more context on spring-cloud-cloudfoundry-discovery. My previous comment related to Spring Cloud Services with Spring Cloud Netflix Eureka, and doesn't apply in the context of spring-cloud-cloudfoundry.

The spring-cloud-cloudfoundry-discovery module doesn't implement registration of apps to a server for lookup by clients. The discovering "client" app calls CF to get a list of all apps in the same CF org and space and creates a mapping of logical app name (e.g. hi-service) to app route (e.g. https://hi-service.apps.mycf.com, https://hi-service-random-words.apps.mycf.com).

To properly support container-to-container networking, there would need to be an option for the discovery logic to map from logical app name to the IP address of each running app instance's container. It doesn't appear that the CF API provides a way to get the in-container IP address of an app instance, only the IP address of the Diego cell that the container is running on and the port for the app instance. Using the Diego cell IP address and instance port is probably enough to bypass the Router and effectively support container-to-container communication.

[making]

Cloud Foundry now supports DNS-based polyglot service discovery.

This sample worked thought it's a naive implementation https://github.com/making/demo-hystrix/blob/master/hystrix-dashboard/src/main/java/com/example/hystrixdashboard/CloudFoundryAppServiceDiscoveryClient.java

How about providing more generic DNS based Discovery Client?

[dsyer]

We're open to ideas. Isn't DNS-based discovery basically how java.net.URL works out of the box?

[making]

Updated the sample to use InetAddress.getAllByName instead of dnsjava

[dsyer]

Looks good. Maybe send a PR? If you can figure out how to write a test that would be awesome.

[making]

Will do. I'll implement followings:

• extend existing CloudFoundryDiscoveryClient and override getService method so that it can inherit getServices.
• rewrite serviceId -> hostname logic using cf-java-client. Currently, i'm assuming that appId equals to hostname but this is not always true.

[scottfrederick]

DNS-based discovery would work fine with newer versions of CF, but isn't supported on older versions. I don't see users asking for container-to-container support with spring-cloud-cloudfoundry right now, so I think going with the future-based DNS approach is fine.

@making One approach would be to add a new property like spring.cloud.cloudfoundry.discovery.useDNS and auto-configure either the existing CloudFoundryDiscoveryClient or a new implementation of the DiscoveryClient based on that property. The DNS-based DiscoveryClient wouldn't need CF Java Client at all (it just needs map appName to appName.apps.internal) so I'm not sure extending the existing CloudFoundryDiscoveryClient makes much sense for DNS-based discovery.

[making]

@scottfrederick @dsyer

I made two candidates. Can you give me advices which one should be appropriate.

• https://github.com/making/spring-cloud-cloudfoundry/commit/f1ee034d69db7433a5e52a477fafbf48e06fe84b
• https://github.com/making/spring-cloud-cloudfoundry/commit/5469b74476d83d61c23920556761fa1e931fb95e

First one is using cf-java-client to retrieve application metadata and checks if internal domain is included in app's urls. index-based routes is used. so we don't need resolve ip address in this client.

Second one is just relying on DNS and does not depends on cf-java-client. I've not tested against DSN server.

[scottfrederick]

@making I haven't looked closely at the code or tested it myself, but did have an observation:

One of the issues I have with the current spring-cloud-cloudfoundry-discovery module is that an app can only "discover" other apps running in the same CF org and space. That's pretty limiting IMO, as apps will often want to interact with apps in other orgs and spaces. I suppose this could be remedied by allowing the discovering apps to configure the names of other orgs/spaces to inspect. You certainly need to constrain it somehow as you wouldn't want to try to inspect all apps on the platform.

It looks like your first option has the same limitation, but the second option does not as it just maps logical app name to an internal route. So that's one consideration.

[spencergibb]

Since they are both configurable, would they be beneficial as is? We're close to an RC1, so if you'd like to submit a PR @making.

[making]

@scottfrederick Container to container networking need network policy (cf add-network-policy frontend --destination-app backend --protocol tcp --port 8080) so I don't think containers can be accessed across spaces.

Supported both type https://github.com/making/spring-cloud-cloudfoundry/commit/dae4a73308db8c43978fb70ada409750f28aa610

I'll send a PR after I test it on PWS.

[nkolosnjaji]

Hello, I want to thank you all for this nice feature, it works as charm. I have one question regarding DNS based discovery: Is it safe to use it in web-flux/reactive application or we need some non-blocking DNS resolver for this one?

[spencergibb]

There shouldn't be any manual DNS resolving happening