Eureka Clustering documentation and best practices

[aivans]

I understand that there is a section in the documentation about Eureka Peer Awareness, but somehow I do not find this enough. I find it hard to understand how this would make Eureka more HA. There would be a chain server1, server2, server3 and they are cross referenced, but if one node fails the chain breaks anyway. Would it be a good option to just specify a list containing all 3 servers separated by comma for each all of them (server1, server2, server3)?

What is your experience?

PS: did not try DNS yet

[aivans]

Here are my fist findings:

• the HA strategy seems to be one main instance with backup eureka servers
• the clients are provided with a list of eureka servers through config (or dns)
• the clients connect to the first server from the list, and this is where they send heartbeats and fetch information
• the other eureka servers are just sitting idle and have no registered application (unless eureka servers reference each other, and in this case only referenced eureka instances will show app, but no business clients)
• in case server1 becomes unavailable, all the clients migrate to the next eureka server from the list and keep trying to contact server1
• in case server1 comes back online, all the clients migrate back to server1

Remarks:

• this seems to be sufficient to have HA considering that clients cache locally

Questions and concerns:

• in case of a high number of clients, it may be that in case server1 fails, server2 may fail too because all clients will try to migrate to it Seems like they took care of this when enabling DNS for serviceURLs: // Rearrange the fail over server list to distribute the load arrangeListBasedonHostname(serviceUrls); This leads to another concern.
• I tried to connect a few services to server1 and a few to server2 by specifying different orders in serviceUrls list. The problem is that in this case topology partitions are created. Even if the 2 eureka servers are cross referenced and they are aware of eachother they do not exchange their registered applications, so BusinessService1 connected to Server1 will have no knowledge about BusinessService2 connected to Server2. The only time the servers have same registry information is when a server is restarted manually. At this point DiscoveryClient.getAndStoreFullRegistry() is called at startup. But after a few minutes, some instances are expired becuase no heartbeats are received.

Remarks:

• would be nice to be able to distribute clients to different Eureka servers if the servers would exchange their registries regularly, but maybe this was not the intention
• Maybe it is working with DNS enabled since there exists the capability to connect to different servers in the list randomly

[aivans]

Hi again,

I am pleased to announce that the behaviour from the previous post happens because all my Eureka nodes were configured with localhost as host name. Changing to peer1 and peer2 as in the documentation allows for the creation of a mirrored style cluster.

[radenui]

Hello,

I have a similar problem here. I'm using AWS, and I'd like to leverage the HA provided by Netflix using DNS and the region settings (https://github.com/Netflix/eureka/wiki/Configuring-Eureka-in-AWS-Cloud#configuring-eips-using-dns). I'm not really interested in EIP or Auto-scaling groups here, just the DNS part.

As my Eureka clients will be in different Availability zones, I will need them to contact the Eureka server in their own availability zone (as a priority, with fallback on other zones in case of a problem). And I still will need my Eureka servers to connect each other (across multiple AZs) and share their configuration.

It seems like I cannot use the same properties defined in Netflix wiki (they are not interpreted by the client). And the way the authentication works (in the URL !!) seems a little restrictive in this case.

Do you have any suggestion ?

Thanks a lot,

Arthur

[aivans]

eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka  

The settings above definitely do the trick for me.

[aivans]

I did not test multiple zones though. Please come back and share your results for your multiple zones setup.

[radenui]

Thank you for your answers!

I started using these properties:

eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka
    region: us-east-1
    availabilityZones:
      us-east-1: us-east-1a,us-east-1b
    preferSameZoneEureka: true

On the DNS side, I have:

• a TXT record for the region
• one TXT record by zone (with "user:password@server" as a value).

Into my Eureka servers (wanted them to descover their peers with DNS) and it seems like the DNS discovery does not allow to define peers at all: I had a very strange and inconsistent behavior using the same client params. Any idea why ?

But this seem to work pretty well with the eureka clients:

For the availability zones, not sure what is happening: I have this error sometimes (and have no idea why):

No match for the zone us-east-1a in the list of available zones [us-east-1b]

And

Updating the serviceUrls as they seem to have changed from [http://user:password@server-us-east-1b:8761/eureka/] to [http://user:password@server-us-east-1a:8761/eureka/]

I will try to increase the log level to see what's happening here, and will keep you in touch. Thanks a lot,

Arthur

[aivans]

Never tested multiple zones. In one single zone (defaultZone) I could configure multiple servers and they saw eachother as replicas.

How are your clients configured?

[radenui]

I tried the same "client" configuration for both eureka clients and servers (see previous message). It only works on the clients. On the Eureka servers, they are not able to use the DNS information to define peers: I had to use "defaultZone" as well.

[aivans]

i would start with the default region(don't try to set it yet) and default zone and see if the dns config is ok. It is ok when in the Eureka web UI you will see servers as available replicas when they are all started. Afterwards, I would try a different zone for one server.

My dns config: txt.us-east-1 IN TXT "defaultZone.eureka.local" txt.defaultZone IN TXT "user:password@peer1" "user:password@peer2"

[aivans]

availabilityZones:
      us-east-1: us-east-1a,us-east-1b

My hunch is that for the client you should specify one single zone, after all the client runs in a single zone, isn't it?

[aivans]

Actually, i think your issue is that you specify availabilityZones. you should leave that to the dns.

[radenui]

Ok, I can try that. But as I cannot set the datacenter as "Amazon", how does the eureka client know its zone ?

[aivans]

datacenter=cloud?

[radenui]

I like that! what is the full name for this property? just "datacenter" ?

[aivans]

Arthur, maybe you need to read the docs a bit, and the source code.

[aivans]

never tried it myself on Amazon

[radenui]

Well, I'll be delighted to find some docs... Because when I believe what I see in the source code, I have this! Thanks for your help Adrian, I'll be interested if you can find any documentation on it.

[ryanjbaxter]

+1 adding additional documentation around how to configure this would be helpful

[aivans]

@ryanjbaxter are you referring to Eureka Clustering in general or Amazon info?

[cforce]

Is there an exampe (including dns records setup) describing how to setup 4 eureka nodes , whre each of two are in an own zone: zone1 and zone2 beeing in the same region but having zone affinity load balancing for all zuul and feign requests?

[Shawn-Cao]

+1 need an example with 3+ nodes.

The sample on http://projects.spring.io/spring-cloud/spring-cloud.html#_peer_awareness doesn't look right. I expect each peer to still default to it's own zone while knowing peers from other zones.

I'm playing with config like below without success:

eureka:
  client:
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
    availability-zones:
      local: http://localhost:8761/eureka/
      local2: http://localhost:8762/eureka/
      local3: http://localhost:8763/eureka/
      cloud: ...

[kennedyoliveira]

+1, i find the @cforce comments interesting, would like to see/know how to do that.

[Shawn-Cao]

After some trial and error. I got a configuration set which allow a few Eureka servers synchronize and share apps registered under each of them. Sharing below:

----eureka-app.yml---- (for eureka servers only)
eureka:
  client:
    prefer-same-zone-eureka: true
    use-dns-for-fetching-service-urls: false
    region: shawn
#    fetch-remote-regions-registry: 'shawn,region2'
    availability-zones:
      shawn: 'pcfShawn,localShawn'
      region2: 'pcf1,pcf2'
    serviceUrl:
      'pcfShawn': 'http://<yourCloudFoundryEurekaUrl>/eureka/'
      'localShawn': 'http://XCao:8761/eureka/'
      'pcf1': 'http://<somewhereElseEurekaUrl>/eureka/'
      'pcf2': 'http://<yetSomewhereElseEurekaUrl>/eureka/'

----application.properties---- (for all apps)
#Eureka
eureka.instance.appname=${spring.application.name}
eureka.instance.non-secure-port=${server.port}
eureka.instance.hostname=${localhost.hostname:<yourHostName>}
eureka.instance.virtualHostName=${eureka.instance.hostname}:${eureka.instance.non-secure-port}
eureka.instance.instanceId=${eureka.instance.virtualHostName}
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka/

[cforce]

This seems incomplete to me. Where is the eureka cfg in the same zone sycned with same zone affinity,where the eureka in the different region configured for region failover. Where is per region the client configuration?

[daggerok]

check it out 1

$ cat /etc/hosts
127.0.0.1       peer1
127.0.0.1       peer2

2

$ cat configs/eureka-replica.properties
server.port=8761
eureka.datacenter=peer1
eureka.instance.hostname=peer1
eureka.client.serviceUrl.defaultZone=http://peer2:8762/eureka/

3

$ cat configs/eureka.properties
server.port=8762
eureka.datacenter=peer2
eureka.instance.hostname=peer2
eureka.client.serviceUrl.defaultZone=http://peer1:8761/eureka/

4

$ cat configs/application.properties
eureka.server.registry-sync-retry-wait-ms=500
eureka.server.a-sgcache-expiry-timeout-ms=60000
eureka.server.eviction-interval-timer-in-ms=30000
eureka.server.peer-eureka-nodes-update-interval-ms=15000
eureka.server.renewal-threshold-update-interval-ms=300000

eureka.client.healthcheck.enabled=true
eureka.client.prefer-same-zone-eureka=true
eureka.client.region=zone2
eureka.client.availability-zones.zone1='peer1,peer2'
eureka.client.availability-zones.zone2='peer2,peer1'
eureka.client.serviceUrl.peer1=http://peer1:8761/eureka/
eureka.client.serviceUrl.peer2=http://peer2:8762/eureka/
eureka.client.serviceUrl.defaultZone=http://peer2:8762/eureka/
# seems like defaultZone must be configured
# so defaultZone peer will use all clients on start up registration if it will not specify
eureka.instance.appname=${spring.application.name}
eureka.instance.instanceId=${spring.application.name}-${spring.cloud.client.ipAddress}-${INSTANCE_ID:${random.value}}
eureka.instance.statusPageUrlPath=${management.contextPath}/info
eureka.instance.healthCheckUrlPath=${management.contextPath}/health
eureka.instance.preferIpAddress=true
# and other out of topic options
management.contextPath=/admin
endpoints.health.sensitive=false
spring.cloud.config.discovery.enabled=true

when not first peer will start, you'll find logs like these:

...PeerAwareInstanceRegistryImpl    : Got <some numbers of> instances from neighboring DS node
...PeerAwareInstanceRegistryImpl    : Renew threshold is: <some other number>
...PeerAwareInstanceRegistryImpl    : Changing status to UP

when peer1 go down, you will see log like this:

ERROR ... [-target_peer1-1] ...cluster.ReplicationTaskProcessor   : Network level connection to peer peer1; retrying after delay

as soon it will restore you'll see common instance registeration log message

[asarkar]

I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.

[streethawk]

I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.

@asarkar do you have the updated link? The one above isn't reachable

[asarkar]

@streethawk Fixed.