Eureka won't properly properly identify members

[nfedyk]

I'm using Spring Cloud Camden.SR2 with Spring Boot 1.4.2.RELEASE running in docker 1.12 with swarm mode enabled. I have ingress network created within swarm.

098cee9f54c1        bridge                           bridge              local
a3686d96ba7f        docker_gwbridge                  bridge              local
2a7e447767eb        host                             host                local
6gdu3f79kyho        ingress                          overlay             swarm
wdvwux5v6t5m        sample-test-network              overlay             swarm
b2220324de65        none                             null                local

When inside application container network looks this way:

bash-4.3# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:FF:00:13
          inet addr:10.255.0.19  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:3905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3694 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:459174 (448.4 KiB)  TX bytes:1440923 (1.3 MiB)

eth1      Link encap:Ethernet  HWaddr 02:42:AC:12:00:05
          inet addr:172.xxx.xxx.5  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:49 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:18338 (17.9 KiB)  TX bytes:4722 (4.6 KiB)

eth2      Link encap:Ethernet  HWaddr 02:42:0A:00:00:12
          inet addr:10.0.0.18  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:962 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1284 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:332468 (324.6 KiB)  TX bytes:210285 (205.3 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:170 errors:0 dropped:0 overruns:0 frame:0
          TX packets:170 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:13096 (12.7 KiB)  TX bytes:13096 (12.7 KiB)

Spring Clioud application is being started with following configuration:

eureka.instance.securePortEnabled: true
eureka.instance.nonSecurePortEnabled: false
eureka.instance.securePort: 30005
eureka.client.serviceUrl.defaultZone: https://user:password@container-id:8080/eureka

eureka.instance.prefer-ip-address: false
spring.cloud.inetutils.preferredNetworks:
    - 10.0

I run 6 services with 2 replicas per service. When I look at /eureka/apps endpoint to check services configuration I see mix of container ids and ip addresses eventhou my config directly states to use names only with prefer-ip-address: false setting.

Sample output

<applications>
  <versions__delta>1</versions__delta>
  <apps__hashcode>UP_12_</apps__hashcode>
  <application>
    <name>SAMPLE-ROUTER</name>
    <instance>
      <hostName>98c739f93aff</hostName>
      <app>SAMPLE-ROUTER</app>
      <ipAddr>10.0.0.18</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
    <instance>
      <hostName>10.255.0.18</hostName>
      <app>SAMPLE-ROUTER</app>
      <ipAddr>10.255.0.18</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
  </application>
  ................................................
  <application>
    <name>SAMPLE-DISCOVERY</name>
    <instance>
      <hostName>3bbcf4a1e714</hostName>
      <app>SAMPLE-DISCOVERY</app>
      <ipAddr>10.0.0.8</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
    <instance>
      <hostName>ceaa7a117f53</hostName>
      <app>SAMPLE-DISCOVERY</app>
      <ipAddr>10.0.0.9</ipAddr>
      <status>UP</status>=
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
  </application>
  <application>
    <name>SAMPLE-SECURITY</name>
    <instance>
      <hostName>10.255.0.13</hostName>
      <app>SAMPLE-SECURITY</app>
      <ipAddr>10.255.0.13</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
    <instance>
      <hostName>10.255.0.12</hostName>
      <app>SAMPLE-SECURITY</app>
      <ipAddr>10.255.0.12</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
  </application>
  <application>
    <name>SAMPLE-CONFIG</name>
    <instance>
      <hostName>3d265cd467e0</hostName>
      <app>SAMPLE-CONFIG</app>
      <ipAddr>10.0.0.5</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
    <instance>
      <hostName>d6ad2791fb77</hostName>
      <app>SAMPLE-CONFIG</app>
      <ipAddr>10.0.0.6</ipAddr>
      <status>UP</status>
      <port enabled="false">8080</port>
      <securePort enabled="true">8080</securePort>
    </instance>
  </application>
</applications>

I wonder why do some services identify using 10.255.0 subnet and improperly register themselve with eureka services. Is there any way to enforce all sevice to abide the preferredNetworks setting?

[dsyer]

Why did you set eureka.instance.prefer-ip-address: false (seems inconsistent with spring.cloud.inetutils.preferredNetworks)?

[nfedyk]

I got it from old issues in this project. This property is being enforced as expected, I will try it in camel cased version as well. I have issue only with spring.cloud.inetutils.preferredNetworks.

[bitsofinfo]

We get the same behavior regardless of eureka.instance.prefer-ip-address: false|true. Some services pick up the ingress network IP, others the 10.0.x.x ip (which is expected). So some services can connect and others can't. It seems wildly inconsistent. Per the example above.

[bitsofinfo]

in eureka discovery (eureka/apps) some nodes list like this

INGRESS ip....<instanceId>10.255.0.6:my-service:8080</instanceId>

while yet others like this

HOSTNAME....<instanceId>63c567e1f4ff:my-service:8080</instanceId>

All of these nodes boot w/

spring.profiles: docker-network-stage
eureka.instance.prefer-ip-address: true
spring.cloud.inetutils.preferredNetworks:
    - 10.0

They are all on the same overlay network, as a swarm service...

[bitsofinfo]

And some instances subsequently with

<hostName>10.255.0.16</hostName>

while others instances for same service with

<hostName>10.0.0.14</hostName> *EXPECTED

We even tried the below ( where eth0 is the ingress swarm network 10.255.x that discovery SHOULD not report)

spring.profiles: docker-network-stage
eureka.instance.prefer-ip-address: true
spring.cloud.inetutils.preferredNetworks:
    - 10.0
spring.cloud.inetutils.ignoredInterfaces:
    - eth0

We effectively cannot run this app on a swarm service across more than a one node swarm cluster because of this. Ideas? Are there other options we are missing here?

[bitsofinfo]

The only way we can get this to work is via system properties

-Deureka.instance.prefer-ip-address=true 
-Dspring.cloud.inetutils.preferredNetworks=10.0 
-Dspring.cloud.inetutils.ignoredInterfaces=eth0

So are these bootstrap properties only? Perhaps we missed this in the docs or its not there. I didn't see it.

The configs above are in the YML file served from the config server

[dsyer]

When are you registering with eureka? Is that happening during bootstrap? E.g. do you need eureka to locate the configserver?

[bitsofinfo]

The config server comes up w/out eureka but does eventually register once the eureka server is up.

2 eureka nodes are started and they register w/ themselves, and get their configs from the config server.

N other application services, register w/ eureka and then discover their config server via eureka, so yes I would assume maybe the issue is that since that config above was previously in an application.yml file (served from config server) it was not taking effect until AFTER discovery registration which would make sense as to why we were seeing that odd random selection of the IP's to pick

It appears that specifying via system props (or I guess it could be hardwired in a bootstrap config w/ the container) then that is avoided and the correct ips are chosen prior to talking to discovery.

Perhaps something more clear in docs could mention this scenario/requirement (i.e. the settings above as boostrap props). Or it is already there and we missed it.

[spencergibb]

Config gathered via config server is to late if you are using eureka to find config server.

[dsyer]

Yeah, it's obvious when you think about it. Where do you think the right place is to add documentation?

[bitsofinfo]

I'm might add a short note here http://cloud.spring.io/spring-cloud-static/spring-cloud.html#ignore-network-interfaces as docker containers are already mentioned. Just a short reminder of setting as a boostrap property vs one fetched via config server (esp if using discovery to find the config server)

[spencergibb]

Closing this due to inactivity. Please re-open if there's more to discuss.