search

codecentric / spring-boot-admin

Admin UI for administration of spring boot applications
Apache License 2.0
12.37k stars 3.08k forks source link

New Instances are not discovered using Eureka #1327

Closed davgia closed 4 years ago

davgia commented 4 years ago

Spring Boot Admin Server do not discover new instances through EurekaDiscoveryClient. It just import the registry just one time (on startup) and it never refreshes again. The problem seems to be the DiscoveryClient, but I was unable to pinpoint the actual problem. It may be a configuration defined by SBA that causes the client to not behave correctly. In fact, this problem occurs only with Spring Boot Admin Server application. I don't think it's a eureka-server problem. In addition, the same Spring Boot Admin Server instance deployed on Kubernetes works fine. I've notices that the getServices() method of the EurekaDiscoveryClient returns always the same list of application ids and it never changes and it's called multiple times, so it's not a scheduling problem either.

Details

Working:

Not Working:

To reproduce the error it's sufficient to start eureka-server (parent: 2.2.1.RELEASE) and Spring Boot Admin Server. SBA should see the eureka-server instance (checking from the UI). After that, any new application registered to eureka-server won't be detected by SBA. Moreover, any initially discovered instance will never be delete from SBA, it will just remain OFFLINE forever.

joshiste commented 4 years ago

Which version of Spring Cloud are you using? Did you figure out that it's not a bug in Spring Cloud Netflix?

SBA refreshes the Instances form discovery every time a new Heartbeat event is published. Could you please try to debug if the event is published by the eureka discovery client?

joshiste commented 4 years ago

May be this is linked: https://github.com/spring-cloud/spring-cloud-netflix/issues/3708

davgia commented 4 years ago

Which version of Spring Cloud are you using?

I'm using Hoxton.RELEASE

Did you figure out that it's not a bug in Spring Cloud Netflix?

I didn't had much time to debug. But I'm pretty sure (but not entirely) that it's not a spring cloud netflix problem because the discovery works fine on any other Spring Application in my project.

SBA refreshes the Instances form discovery every time a new Heartbeat event is published. Could you please try to debug if the event is published by the eureka discovery client?

I've change the logging level of the InstanceDiscoveryListener and as far as I can see the Heartbeat events are received (I see Discovering new instances from DiscoveryClient print repeatedly).

Moreover, what I have done to test it is to add a new java class in the project that autowires the EurekaDiscoveryClient and schedule a method that calls the getServices() and prints the result. I can clearly see that the discovery client is not receiving any new instance while on eureka they are correctly registered.

May be this is linked: spring-cloud/spring-cloud-netflix#3708

I think I've never seen the error, but it could be related. This may explain why the first time the registry is fetched correctly. I will look into it and let you know.

bberto commented 4 years ago

Hi @DaviGia, I'm the author of spring-cloud/spring-cloud-netflix#3708. Are you using bootstrap.yml when you reproduce the issue?

However I tested also without Spring Boot Admin, and the issue remains (looking at /actuator/health/discoveryComposite)

davgia commented 4 years ago

Hi @bberto, I am in fact using a bootstrap.yml configuration. It's not the same as yours but I think I too use eureka to discover config-server instance. I will try to disable config-server discovery and see if it works (it may take a while).

davgia commented 4 years ago

/actuator/health/discoveryComposite

Hi @bberto, I tried to access this route but it gives me 404. Is it a standard actuator?

Anyway, this is the content of the bootstrap.yml used by my SBA instance:

spring:
  application:
    name: springboot-admin
  cloud:
    kubernetes:
      enabled: false
    config:
      fail-fast: true
      discovery:
        enabled: true
        serviceId: config-server

eureka:
  client:
    registerWithEureka: false
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/

I can confim that if I disable cloud config discovery (spring.cloud.config.discovery: false) SBA can indeed discover new instances. So it's not a problem related to SBA. I will close this issue. Thanks for the feedback.

bberto commented 4 years ago

It's a standard actuator endpoint but it's an health detail, so you should enable it like this:

management:
  endpoints:
    enabled-by-default: true
    web.exposure.include: "*"
  endpoint:
    health:
      show-details: always

I found the issue has been further analyzed here spring-cloud/spring-cloud-netflix#3675 and reported also on spring-config side.

davgia commented 4 years ago

Thanks for the tip. And yes, I've too discovered the #3675 issue. I will continue to keep an eye on it.