search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

Feign Client Load Balancer not able to reach service on a remote server #2776

Closed VDigavalli closed 5 years ago

VDigavalli commented 6 years ago

Feign Client not working as expected. I have a Eureka Server and Config server running on "tdcwstd3va005.eu.company.com" server.

I have two client service applications (EmailService, EmailServiceServer) running on "tdcwstd3va006.eu.company.com" server.

EmailServiceServer is using @FeignClient to access EmailService.

Below is Eureka Server config property: application.yml:

server: port: 8010

eureka: instance: hostname: localhost client: registerWithEureka: false fetchRegistry: false serviceUrl: defaultZone: http://localhost:8010/eureka/

Below is EmailService property:

application.yml:

server: port: 8028 eureka: client: registerWithEureka: true fetchRegistry: true serviceUrl: defaultZone: http://tdcwstd3va005.eu.company.com:8761/eureka/

Below is EmailServiceServer property:

application.yml:

server: port: 8029 eureka: client: registerWithEureka: true fetchRegistry: true serviceUrl: defaultZone: http://tdcwstd3va005.eu.company.com:8761/eureka/

I dont see any issues when Both the client apps are running on the same server "tdcwstd3va006.eu.company.com".

But once I start the "EmailService" service on "tdcwstd3va005.eu.company.com" and stop the one of "tdcwstd3va005.eu.company.com", I get below exception in the http response:

{"timestamp":"2018-03-13T22:54:01.691+0000","status":500,"error":"Internal Server Error","message":"tdcwstd3va005.uswin.ad.companycorp.com executing POST http://EMAILSERVICE/sendEmail","path":"/sendEmail"}

Note: you can see the server hostname URL is changed in the response from "eu.company.com" to "uswin.ad.company.com"

Below is the exception from EmailServiceServer log files:

2018-03-13 18:49:51.098 INFO 7476 --- [http-nio-8030-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet' 2018-03-13 18:49:51.099 INFO 7476 --- [http-nio-8030-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started 2018-03-13 18:49:51.133 INFO 7476 --- [http-nio-8030-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 34 ms 2018-03-13 18:49:51.229 INFO 7476 --- [http-nio-8030-exec-1] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@69492c22: startup date [Tue Mar 13 18:49:51 EDT 2018]; parent: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@4cf4d528 2018-03-13 18:49:51.306 INFO 7476 --- [http-nio-8030-exec-1] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2018-03-13 18:49:51.685 INFO 7476 --- [http-nio-8030-exec-1] c.netflix.config.ChainedDynamicProperty : Flipping property: EMAILSERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 2018-03-13 18:49:51.719 INFO 7476 --- [http-nio-8030-exec-1] c.n.u.concurrent.ShutdownEnabledTimer : Shutdown hook installed for: NFLoadBalancer-PingTimer-EMAILSERVICE 2018-03-13 18:49:51.775 INFO 7476 --- [http-nio-8030-exec-1] c.netflix.loadbalancer.BaseLoadBalancer : Client: EMAILSERVICE instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=EMAILSERVICE,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null 2018-03-13 18:49:51.786 INFO 7476 --- [http-nio-8030-exec-1] c.n.l.DynamicServerListLoadBalancer : Using serverListUpdater PollingServerListUpdater 2018-03-13 18:49:52.040 INFO 7476 --- [http-nio-8030-exec-1] c.netflix.config.ChainedDynamicProperty : Flipping property: EMAILSERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 2018-03-13 18:49:52.043 INFO 7476 --- [http-nio-8030-exec-1] c.n.l.DynamicServerListLoadBalancer : DynamicServerListLoadBalancer for client EMAILSERVICE initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=EMAILSERVICE,current list of Servers=[tdcwstd3va006.uswin.ad.vzwcorp.com:8029],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;] },Server stats: [[Server:tdcwstd3va006.uswin.ad.vzwcorp.com:8029; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Wed Dec 31 19:00:00 EST 1969; First connection made: Wed Dec 31 19:00:00 EST 1969; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0] ]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@f476b00 2018-03-13 18:49:52.791 INFO 7476 --- [PollingServerListUpdater-0] c.netflix.config.ChainedDynamicProperty : Flipping property: EMAILSERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 2018-03-13 18:52:55.529 ERROR 7476 --- [http-nio-8030-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is feign.RetryableException: connect timed out executing POST http://EMAILSERVICE/sendEmail] with root cause

java.net.SocketTimeoutException: connect timed out at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_161] at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_161] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_161] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_161] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_161] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_161] at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_161] at feign.Client$Default.convertAndSend(Client.java:133) ~[feign-core-9.5.1.jar!/:na] at feign.Client$Default.execute(Client.java:73) ~[feign-core-9.5.1.jar!/:na] at org.springframework.cloud.netflix.feign.ribbon.FeignLoadBalancer.execute(FeignLoadBalancer.java:80) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at org.springframework.cloud.netflix.feign.ribbon.FeignLoadBalancer.execute(FeignLoadBalancer.java:48) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:104) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) ~[rxjava-1.3.4.jar!/:1.3.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Subscriber.setProducer(Subscriber.java:209) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10352) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10319) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[rxjava-1.3.4.jar!/:1.3.4] at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97) ~[feign-core-9.5.1.jar!/:na] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76) ~[feign-core-9.5.1.jar!/:na] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103) ~[feign-core-9.5.1.jar!/:na] at com.sun.proxy.$Proxy118.sendEmail(Unknown Source) ~[na:na] at com.eits.emailservice.DoWork.doSomeWork(DoWork.java:13) ~[classes!/:1.0] at com.eits.emailservice.EmailServiceController.sendEmail(EmailServiceController.java:31) ~[classes!/:1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:871) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:777) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:119) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterWithMetrics(WebMvcMetricsFilter.java:95) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:72) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

After a certain while, I see the below exceptions in the log file when hitting EmailServiceServer : 2018-03-13 18:53:52.470 INFO 7476 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration 2018-03-13 18:53:55.474 WARN 7476 --- [http-nio-8030-exec-6] c.netflix.loadbalancer.BaseLoadBalancer : LoadBalancer [EMAILSERVICE]: Error choosing server for key null

java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310) ~[guava-18.0.jar!/:na] at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:292) ~[guava-18.0.jar!/:na] at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) ~[guava-18.0.jar!/:na] at com.netflix.loadbalancer.AbstractServerPredicate.chooseRoundRobinAfterFiltering(AbstractServerPredicate.java:203) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.PredicateBasedRule.choose(PredicateBasedRule.java:45) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.BaseLoadBalancer.chooseServer(BaseLoadBalancer.java:736) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.ZoneAwareLoadBalancer.chooseServer(ZoneAwareLoadBalancer.java:113) [ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:481) [ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184) [ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) [ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) [rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276) [rxjava-1.3.4.jar!/:1.3.4] at rx.Subscriber.setProducer(Subscriber.java:209) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) [rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) [rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10352) [rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10319) [rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) [rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.single(BlockingObservable.java:340) [rxjava-1.3.4.jar!/:1.3.4] at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) [ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63) [spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97) [feign-core-9.5.1.jar!/:na] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76) [feign-core-9.5.1.jar!/:na] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103) [feign-core-9.5.1.jar!/:na] at com.sun.proxy.$Proxy118.sendEmail(Unknown Source) [na:na] at com.eits.emailservice.DoWork.doSomeWork(DoWork.java:13) [classes!/:1.0] at com.eits.emailservice.EmailServiceController.sendEmail(EmailServiceController.java:31) [classes!/:1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:871) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:777) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:119) [spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterWithMetrics(WebMvcMetricsFilter.java:95) [spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:72) [spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

2018-03-13 18:53:55.478 ERROR 7476 --- [http-nio-8030-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: EMAILSERVICE] with root cause

com.netflix.client.ClientException: Load balancer does not have available server for client: EMAILSERVICE at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:483) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Subscriber.setProducer(Subscriber.java:209) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10352) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10319) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[rxjava-1.3.4.jar!/:1.3.4] at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97) ~[feign-core-9.5.1.jar!/:na] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76) ~[feign-core-9.5.1.jar!/:na] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103) ~[feign-core-9.5.1.jar!/:na] at com.sun.proxy.$Proxy118.sendEmail(Unknown Source) ~[na:na] at com.eits.emailservice.DoWork.doSomeWork(DoWork.java:13) ~[classes!/:1.0] at com.eits.emailservice.EmailServiceController.sendEmail(EmailServiceController.java:31) ~[classes!/:1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:871) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:777) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:119) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterWithMetrics(WebMvcMetricsFilter.java:95) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:72) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

2018-03-13 18:54:01.689 ERROR 7476 --- [http-nio-8030-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is feign.RetryableException: tdcwstd3va005.uswin.ad.vzwcorp.com executing POST http://EMAILSERVICE/sendEmail] with root cause

java.net.UnknownHostException: tdcwstd3va005.uswin.ad.vzwcorp.com at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_161] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_161] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_161] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_161] at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_161] at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_161] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_161] at feign.Client$Default.convertAndSend(Client.java:133) ~[feign-core-9.5.1.jar!/:na] at feign.Client$Default.execute(Client.java:73) ~[feign-core-9.5.1.jar!/:na] at org.springframework.cloud.netflix.feign.ribbon.FeignLoadBalancer.execute(FeignLoadBalancer.java:80) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at org.springframework.cloud.netflix.feign.ribbon.FeignLoadBalancer.execute(FeignLoadBalancer.java:48) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:104) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) ~[rxjava-1.3.4.jar!/:1.3.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.unsafeSubscribe(Observable.java:10256) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Subscriber.setProducer(Subscriber.java:209) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10352) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.Observable.subscribe(Observable.java:10319) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) ~[rxjava-1.3.4.jar!/:1.3.4] at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[rxjava-1.3.4.jar!/:1.3.4] at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) ~[ribbon-loadbalancer-2.2.4.jar!/:2.2.4] at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63) ~[spring-cloud-netflix-core-2.0.0.M5.jar!/:2.0.0.M5] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97) ~[feign-core-9.5.1.jar!/:na] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76) ~[feign-core-9.5.1.jar!/:na] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103) ~[feign-core-9.5.1.jar!/:na] at com.sun.proxy.$Proxy118.sendEmail(Unknown Source) ~[na:na] at com.eits.emailservice.DoWork.doSomeWork(DoWork.java:13) ~[classes!/:1.0] at com.eits.emailservice.EmailServiceController.sendEmail(EmailServiceController.java:31) ~[classes!/:1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:871) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:777) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:119) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterWithMetrics(WebMvcMetricsFilter.java:95) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:72) ~[spring-boot-actuator-2.0.0.M7.jar!/:2.0.0.M7] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.2.RELEASE.jar!/:5.0.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) ~[tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar!/:8.5.23] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Please let me know what needs to be changed to make this work.

spencergibb commented 6 years ago

Please learn how to properly format code and logs.

Can you provide a complete, minimal, verifiable sample that reproduces the problem?

VDigavalli commented 6 years ago

Apologies. Services.zip

I will make sure I format code and logs properly. I am attaching the Config Server, Eureka Server, tdm_email, tdm_email_server services.

Steps to reproduce:

  1. Start Eureka and Config server on the same box (Server A)
  2. Start tdm_email, tdm_email_server on a different box (Server B)
  3. Send any http POST request with some body to tdm_email_server. You should get a proper response
  4. Start tdm_email service on the box Eureka and Config servers are running (Server A)
  5. Stop tdm_email service on the original box (Server B)
  6. Send any http POST request with some body to tdm_email_server.You should get an error in the response

Please let me know if anything else is needed

VDigavalli commented 6 years ago

@spencergibb . Also, my issue might be (just guessing) that the actual server hostname the services are running is "tdcwstd3va005.eu.company.com". But I see the hostnames as "tdcwstd3va005..uswin.ad.companycorp.com" in Eureka dashboard. Since Eureka is providing the wrong hostnames to Feign when it needs, it might be failing too. Can someone please let me know how to change the URL used to register with Eureka.

ryanjbaxter commented 6 years ago

Your example is a little complex to run (requiring two different servers).

Can someone please let me know how to change the URL used to register with Eureka.

You can set the hostname the Eureka client registers with by setting eureka.instance.hostname.

VDigavalli commented 6 years ago

@ryanjbaxter . Thanks for providing the details. I tried adding below property to the application.yml file for the service.

eureka: instance: hostname: tdcwstd3va006.eu.company.com

But I still see the service getting registered with the hostname "tdcwstd3va006.uswin.ad.company.com" in Eureka.

Also, below is the output of the ipconfig /all command on the server this service is running

H:>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : tdcwstd3va006 Primary Dns Suffix . . . . . . . : uswin.ad.company.com Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : ebiz.company.com company.com itcent.ebiz.company.com ittpa.ebiz.company.com

Do you think its because of the Primary Dns Suffix set to that. Is there a way we can override this.

spencergibb commented 6 years ago

What does http://<eureka server>/eureka/apps say

VDigavalli commented 6 years ago

@spencergibb I see the same tdcwstd3va006.uswin.ad.company.com there as well

ryanjbaxter commented 6 years ago

Could you provide all of the output?

VDigavalli commented 6 years ago

@ryanjbaxter attached is the output Eureka_Output.txt

ryanjbaxter commented 6 years ago

The information in that attachment doesnt seem to correlate to your description of your problem, the service names dont exist.

VDigavalli commented 6 years ago

@ryanjbaxter Apologies. I got the wrong file. Please find attached for the right one Eureka_Services_Output.txt

VDigavalli commented 6 years ago

Also, I tried registering the service manually by sending an API to Eureka with the correct hostname address as below:

{ "instance": { "hostName": "tdcwstd3va006.eu.company.com", "app": "EMAILSERVICE", "vipAddress": "tdcwstd3va006.eu.company.com", "secureVipAddress": "tdcwstd3va006.eu.company.com", "ipAddr": "10.144.51.1", "status": "UP", "port": {"$": "8028", "@enabled": "true"}, "securePort": {"$": "8028", "@enabled": "true"}, "healthCheckUrl": "http://tdcwstd3va006.eu.company:8028/healthcheck", "statusPageUrl": "http://tdcwstd3va006.eu.company:8028/status", "homePageUrl": "http://tdcwstd3va006.eu.company:8028", "dataCenterInfo": { "@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo", "name": "MyOwn" } } }

I did see the service showing up in the Eureka dashboard after that with the correct host name address.

Added below property in the EMAILSERVICE application.yml file

server: port: 8028 eureka: client: registerWithEureka: false fetchRegistry: true serviceUrl: defaultZone: http://tdcwstd3va005.eu.company.com:8080/EurekaServer/eureka/

I added similar properties for EMAILSERVICESERVER and registered that service as well manually with Eureka.

But when I sent a request to EMAILSERVICESERVER, it gave me exception: com.netflix.client.ClientException: Load balancer does not have available server for client: EMAILSERVICE

Will this approach work and is there anything else that I need to take care of in the request to Eureka or in properties

spencergibb commented 6 years ago

eureka: instance: hostname: tdcwstd3va006.eu.company.com

But I still see the service getting registered with the hostname "tdcwstd3va006.uswin.ad.company.com" in Eureka.

Something else is going on. Eureka doesn't do anything with the hostname.

spencergibb commented 5 years ago

Closing this due to inactivity. Please re-open if there's more to discuss.