search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

Status Page and Health Indicator paths are broken if configuring custom management context path and using service inside docker #2804

Closed ingogriebsch closed 6 years ago

ingogriebsch commented 6 years ago

We are facing some problems with custom configured status page and health indicator paths.

We are using Spring Boot 1.5.10 and Spring Cloud Edgware.SR3. We are using a Spring Boot application annotating @EnableEurekaServer as the service registry. We are using a Spring Boot application annotating @EnableDiscoveryClient acting as a business service. We have configured the business service the following way:

management:
  context-path: /admin
eureka:
  instance:
    health-check-url-path: ${management.context-path}/health
    status-page-url-path: ${management.context-path}/info

If we are starting up the services locally the business service registers to the service registry and is then accessible through the service registry UI. We also can access the business service management paths directly through a browser or postman like we have configured them. If we are running the service registry in debug mode we can see that the following informations are send:

"homePageUrl":"http://169.254.77.127:33800/"
"statusPageUrl":"http://169.254.77.127:33800/admin/info"
"healthCheckUrl":"http://169.254.77.127:33800/admin/health"

Means all is fine if running the services locally.

But if we are starting up the business service through docker the business service is registering to the eureka service in a way that the paths are looking like the following:

"homePageUrl":"http://169.254.77.127:33800/"
"statusPageUrl":"http://169.254.77.127:8080/admin/admin/info"
"healthCheckUrl":"http://169.254.77.127:8080/admin/admin/health"

As one can see, the path is in some way duplicated and the port is not matching the port of the home page url. If the service registry is running also in docker or locally is not making a difference. We are configuring the business service inside the docker-compose the following way:

business-service:
    environment:
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: $EUREKA_URL
      EUREKA_INSTANCE_HOSTNAME: $IP
      JAVA_TOOL_OPTIONS: -Xmx128m -Deureka.instance.preferIpAddress=false -Deureka.instance.nonSecurePort=33800
      SERVER_PORT: 8080
    ports:
    - 33800:8080

If we are changing the configuration the following way we can solve the problem with the different ports, but it is not solving the problem with the broken path:

business-service:
    environment:
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: $EUREKA_URL
      EUREKA_INSTANCE_HOSTNAME: $IP
      JAVA_TOOL_OPTIONS: -Xmx128m -Deureka.instance.preferIpAddress=false 
      SERVER_PORT: 33800
    ports:
    - 33800:33800

We are more or less sure that we are still understanding and therefore doing something wrong. But playing further with the docker configuration did not solve the problem.

But... If we are switching (back) to Spring Cloud Dalston.SR5 all is fine and the services are working as expected (locally and through docker).

spencergibb commented 6 years ago

When you say "the services are working as expected", you mean, the links on the eureka dashboard work (as those URLs aren't used anywhere else)?

I would expect SERVER_PORT to always be used in setting the urls. Any more configuration you can show? I don't see how we would get 33800 by default ever.

Any chance you can provide a complete, minimal, verifiable sample that reproduces the problem? We don't do anything specific for docker, so it seems strange that using it would make a difference.

batarayappa commented 6 years ago

Hi Spencer,

I have same problem. We are getting 404 for zuul health check url in admin ui and it is working for some of the applications(config server shown below) with Edgware.SR3 and springboot=1.5.10.RELEASE. we are seeing this issue after we migrated to Edageware sr3 from Dalston . Please find below application.yml used by both config server and zuul. Please help me to resolve this issue.

Thanks in advance.

application.yml

management.context-path: '/admin' eureka: instance: hostname: ${hostname} statusPageUrlPath: ${management.context-path}/info healthCheckUrlPath: ${management.context-path}/health

admin-service.yml( used by admin service module only)

spring.boot.admin.discovery.converter.management-context-path: /admin

Zuul access logs

[15/May/2018:11:39:34 -0500] 127.0.0.1 [,-] - - - - - - - - - - - /admin/admin/health "GET /admin/admin/health HTTP/1.1" 404 - 44 "-" "Apache-HttpClient/4.5.5 (Java/1.8.0_151)" "-"

config Server access log

[15/May/2018:11:39:33 -0500] 127.0.0.1 [,-] - - - - - - - - - - - /admin/admin/health "GET /admin/admin/health HTTP/1.1" 200 1718 122 "-" "Apache-HttpClient/4.5.5 (Java/1.8.0_151)" "-"

Harinath.

ryanjbaxter commented 6 years ago

What is the "admin ui"?

batarayappa commented 6 years ago

Hi Ryan, It is Admin server from where we can check all applications health status, environment variables etc.. Please below admin server dependencies for same.

        "de.codecentric:spring-boot-admin-server:1.5.1",
        "de.codecentric:spring-boot-admin-server-ui:1.5.1",
        "de.codecentric:spring-boot-admin-server-ui-login:1.5.1"

Harinath.

spencergibb commented 6 years ago

We don't maintain that. If there is doubling of /admin, why not remove the management.context-path from statusPageUrlPath: ${management.context-path}/info?

batarayappa commented 6 years ago

Thanks for your reply, Spencer.

I have got one more problem when i'm trying to deploy my Zuul in one of the environment. Please suggest suggest me some idea to find out root cause.

com.netflix.zuul.exception.ZuulException: Failed to instantiate [org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient]: Is it an abstract class?; nested exception is java.lang.InstantiationException: org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient
                at org.springframework.cloud.netflix.zuul.util.ZuulRuntimeException.<init>(ZuulRuntimeException.java:33)
                at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:119)
                at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
                at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
                at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
                at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
                at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
                at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
                at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
                at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
                at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
                at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
                at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
                at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
                at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
                at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
                at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:166)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
                at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient]: Is it an abstract class?; nested exception is java.lang.InstantiationException: org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient
                at org.springframework.beans.BeanUtils.instantiate(BeanUtils.java:80)
                at org.springframework.cloud.netflix.ribbon.SpringClientFactory.instantiateWithConfig(SpringClientFactory.java:94)
                at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:115)
                at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClient(SpringClientFactory.java:51)
                at org.springframework.cloud.netflix.zuul.filters.route.apache.HttpClientRibbonCommandFactory.create(HttpClientRibbonCommandFactory.java:54)
                at org.springframework.cloud.netflix.zuul.filters.route.apache.HttpClientRibbonCommandFactory.create(HttpClientRibbonCommandFactory.java:33)
                at org.springframework.cloud.sleuth.instrument.zuul.TraceRibbonCommandFactory.create(TraceRibbonCommandFactory.java:48)
                at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:156)
                at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:111)
                ... 81 common frames omitted
Caused by: java.lang.InstantiationException: org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient
                at java.lang.Class.newInstance(Class.java:427)
                at org.springframework.beans.BeanUtils.instantiate(BeanUtils.java:77)
                ... 89 common frames omitted
Caused by: java.lang.NoSuchMethodException: org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.<init>()
                at java.lang.Class.getConstructor0(Class.java:3082)
                at java.lang.Class.newInstance(Class.java:412)
                ... 90 common frames omitted
spencergibb commented 6 years ago

@batarayappa that is unreleated to this issue. please ask questions on stack overflow or gitter.

ingogriebsch commented 6 years ago

@spencergibb Sorry for the late reply. I will share some additional observations and a test project shortly.

gbtec-ext-sergeyuzhakov commented 6 years ago

Demo project was created to show the issue. https://github.com/gbtec-ag/health-indicator-dalston2edgware-issue By switching

Dalston.SR5 to Edgware.RELEASE in the pom.xml one can notice in http://localhost:8761/eureka/apps that http://127.0.0.1:22520/admin/health changes to http://127.0.0.1:22520/admin/admin/health
gbtec-ext-sergeyuzhakov commented 6 years ago

In 1.4.0 spring-cloud-netflix-eureka-client DefaultManagementMetadataProvider was added. It seems the root cause for the issue. DefaultManagementMetadataProvider#get() returns /admin/admin/health if server.port != 0

ryanjbaxter commented 6 years ago

The more I think about this the more I think the we really just need to document that there is no longer any need to include the management context path in the health check or status url properties and that spring cloud netflix will take care of adding it if it is set.

ingogriebsch commented 6 years ago

@ryanjbaxter @spencergibb We would like to ask if some bug fix is visible on the horizon. And we would like to know if we can help in some way to understand/solve the problem. :)

ryanjbaxter commented 6 years ago

I am going to label this as for team discussion since I think we just need to add some documentation

ingogriebsch commented 6 years ago

@ryanjbaxter @spencergibb I would like to bring up this issue again because you marked it as 'documentation' but I don't understand how this can solve our problem.

As @gbtec-ext-sergeyuzhakov mentioned here it looks like class DefaultManagementMetadataProvider is doing some stuff which breaks the expected behavior.

Maybe we are understand it wrong and configuring our system in a different way solves the problem. But if this is the case and if this is what you want to document we are really keen to see and try to understand the documentation you are about to enhance.

Otherwise it would be really nice if you could have a second look if the code is really doing what you think it should do.

Noxaro commented 6 years ago

I've encountered the same issue with the DefaultManagementMetadataProvider when i'm using a different external docker port. For example the application runs with port 8080 inside the docker container with a docker port mapping to port 80. Therefore i've set the "nonSecurePort" to 80. Then the application will be registered with a wrong actuator ports inside eureka. For example "localhost:8080/actuator/health" instead of "localhost:80/actuator/health".

My personal workaround for this problem was to null the DefaultManagementMetadataProvider in the spring context and let the eureka client do the configuration stuff which is working as expected.

ryanjbaxter commented 6 years ago

@gbtec-ingogriebsch my understanding is that you were adding management.context-path to the health check and status page URL paths in Dalston.X but when using Edgware.SR3 with the same configuration the management context path was being added twice. This was due to new code in Edgware setting the management context path in the URLs automatically AND you also setting it in the health check and status page URL. The reason I labeled this as documentation was to document the fact that setting the management context path in the health check and status page URL is no longer required.

ryanjbaxter commented 6 years ago

Closed via ab7b0fb

gbtec-ext-sergeyuzhakov commented 6 years ago

Hello Ryan. It seems your solution "... setting the management context path in the health check and status page URL is no longer required" doesn't work.

You can check it here https://github.com/gbtec-ag/health-indicator-dalston2edgware-issue

ryanjbaxter commented 6 years ago

What do you mean does not work?

gbtec-ext-sergeyuzhakov commented 6 years ago

I mean this config works only when server.port != 0

eureka:
  instance:
     health-check-url-path: /health
and
management:
   context-path: /admin
and
server:
   port: 0

if server.port == 0 then healthCheckUrl doesn't respect management:context-path and I get

http://IP:56287/health
ryanjbaxter commented 6 years ago

That seems to be a different issue than what was originally reported here, please open a separate issue.