spring-cloud / spring-cloud-consul

Spring Cloud Consul
http://cloud.spring.io/spring-cloud-consul/
Apache License 2.0
813 stars 541 forks source link

Inconsistent FeignClient failover behavior #106

Closed sowston closed 8 years ago

sowston commented 8 years ago

I am witnessing inconsistent failover behavior of the loadbalancer in destructive testing of a REST service that I'm hitting from my FeignClient.

I have a 5 node Consul cluster running in vagrant:

I have deployed a random number web service and registered the service on N3, N4, N5.

I have a SpringBoot application with a FeignClient with autowired LoadBalancer and Consul DiscoveryClient.

The load balancer is working properly when all three services are "UP". If I hard kill a web service node, for example (service randomnumber stop) on the N3 node, I see the the FeignClient properly failover only a couple requests, then eventually it begins to throw this exception:

2015-10-31 12:09:16.058 ERROR 46312 --- [nio-8081-exec-3] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request
processing failed; nested exception is java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: RandomNumber] with root cause

com.netflix.client.ClientException: Load balancer does not have available server for client: RandomNumber
        at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:468)
        at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184)
        at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
        at rx.Observable$1.call(Observable.java:144)
        at rx.Observable$1.call(Observable.java:136)
        at rx.Observable$1.call(Observable.java:144)
        at rx.Observable$1.call(Observable.java:136)
        at rx.Observable$1.call(Observable.java:144)
        at rx.Observable$1.call(Observable.java:136)
        at rx.Observable$1.call(Observable.java:144)
        at rx.Observable$1.call(Observable.java:136)
        at rx.Observable.subscribe(Observable.java:7621)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:442)
        at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
        at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
        at feign.ribbon.RibbonClient.execute(RibbonClient.java:69)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:92)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:71)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:94)
        at com.sun.proxy.$Proxy79.getRandomNumber(Unknown Source)
        at com.viasat.services.rest.randomnumber.RandomNumberClient.getRandomNumber(RandomNumberClient.java:107)
        at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:295)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:68)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)

RandomNumberClient.java

package com.viasat.services.rest.randomnumber;

import java.util.List;
import java.util.Random;

import lombok.extern.slf4j.Slf4j;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.client.loadbalancer.LoadBalancerClient;
import org.springframework.cloud.netflix.feign.EnableFeignClients;
import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;

/**
 * @author Spencer Gibb
 */
@Configuration
//@RibbonClient(name="RandomNumberClient", configuration=RandomNumberClient.class)
@EnableAutoConfiguration
@EnableDiscoveryClient
@RestController
@EnableConfigurationProperties
@EnableFeignClients
@Slf4j
@EnableCircuitBreaker
public class RandomNumberClient {

    @Autowired
    private LoadBalancerClient loadBalancer;

    @Autowired
    private DiscoveryClient discoveryClient;

    @Autowired
    private Environment env;

    @Autowired
    private RandomNumberFeignClient randomNumberFeignClient;

    @Value("${spring.application.serviceurl:RandomNumberClient")
    private String serviceURL;

    @Value("${spring.application.name:RandomNumberClient}")
    private String appName;

    @RequestMapping("/me")
    public ServiceInstance me() {
        return discoveryClient.getLocalServiceInstance();
    }

    @RequestMapping("/")
    public ServiceInstance lb() {
        return loadBalancer.choose(appName);
    }

    @RequestMapping("/choose")
    public String choose() {
        return loadBalancer.choose(appName).getUri().toString();
    }

    @RequestMapping("/chooseservice/{service}")
    public String choose(@PathVariable("service") String service) {
        return loadBalancer.choose(service).getUri().toString();
    }

    @RequestMapping("/myenv")
    public String env(@RequestParam("prop") String prop) {
        return env.getProperty(prop, "Not Found");
    }

    @RequestMapping("/prop")
    public String prop() {
        return sampleProperties().getProp();
    }

    @RequestMapping("/instances/{service}")
    public List<ServiceInstance> instances(@PathVariable("service") String service) {
        System.out.println("ServiceName: " + service);
        return discoveryClient.getInstances(service);
    }

    @RequestMapping("/feign")
    public String feign() {
        return randomNumberFeignClient.choose();
    }

    @RequestMapping("/services/rest/randomnumber")
    public String getRandomNumber() {
        return randomNumberFeignClient.getRandomNumber();
    }

    @RequestMapping("/service/rest/randomnumber")
    public String randomnumber() {
        return getRandomNumber();
    }

    public String fallback() {
        Random randomGenerator = new Random();
        Integer rn = randomGenerator.nextInt();
        return rn.toString();   
    }

    /*@Bean
    public SubtypeModule sampleSubtypeModule() {
        return new SubtypeModule(SimpleRemoteEvent.class);
    }*/

    @Bean
    public SampleProperties sampleProperties() {
        return new SampleProperties();
    }

    public static void main(String[] args) {
        SpringApplication.run(RandomNumberClient.class, args);
    }

    /*@Override
    public void onApplicationEvent(SimpleRemoteEvent event) {
        log.info("Received event: {}", event);
    }*/

    @FeignClient("RandomNumber")
    public interface RandomNumberFeignClient {

        @RequestMapping(value = "/choose", method = RequestMethod.GET)
        String choose();

        @RequestMapping(value = "/services/rest/randomnumber", method = RequestMethod.GET)
        String getRandomNumber();
    }
}

bootstrap.yml

spring:
  application:
    name: RandomNumberClient
    port: 8081
  cloud:
    consul:
      host: 172.20.20.10
      port: 8500

application.yml

server:
  port: 8081

RandomNumberClient:
  ribbon: 
    UseIPAddrForServe: true
    ServerListRefreshInterval: 60000
    VipAddress: RandomNumberClient
    MaxAutoRetries: 1
    OkToRetryOnAllOperations: true
    NIWSServerListClassName: com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList

endpoints:
  health:
    sensitive: false
  restart:
    enabled: true
  shutdown:
    enabled: true

logging:
  level:
    org.springframework.cloud.consul: DEBUG
spencergibb commented 8 years ago

This is a setting for eureka NIWSServerListClassName: com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList.

spencergibb commented 8 years ago

Do you have a service with spring.application.name=RandomNumber?

spencergibb commented 8 years ago

I am unable to reproduce. Do you have a project to share?

sowston commented 8 years ago

Yes, I have an entire workspace I can share. What is the best way to share it?

sowston commented 8 years ago

Yes, I have a service "RandomNumber" Service, it's a REST service with one "randomnumber" resource that supports GET. I will send it forward with the workspace to u. Can also provide a vagrant workspace to spin up the env.

spencergibb commented 8 years ago

@sowston however you want to get it to me. Dropbox, google drive, email, whatever.

sowston commented 8 years ago

Here's the Google Drive link to the RandomNumber REST Web Service Project: https://drive.google.com/file/d/0B5e-22Pji8-_bDRLR0xwMUlKMnc/view?usp=sharing

Here's the link to my SpringCloud Client Project: https://drive.google.com/file/d/0B5e-22Pji8-_czV5T0RXN2gwclk/view?usp=sharing

Here's the vagrant env files: https://drive.google.com/file/d/0B5e-22Pji8-_Z2w1Zkc4cG8wNVU/view?usp=sharing

and thanks!

sowston commented 8 years ago

@spencergibb if there is a chance for you to look at this issue, I would be most appreciative. I can't seem to get around the issue.

sowston commented 8 years ago

also, here's the service.json to configure consul with the RandomNumber service:

randomnumberservice.json node 3:

{

  "service": {

    "name": "RandomNumber",

    "tags": ["stateless"],

    "address": "172.20.20.30",

    "port": 8080
,
    "checks": [
      {
        "id": "RandomNumberRESTGetCheck",
        "http": "http://localhost:8080/services/rest/health",
        "interval": "3s",
        "timeout": "1s"
      }
    ]
  }

}

node 4 and 5 have the same config, with address: .40 and .50 respectively.

spencergibb commented 8 years ago

hmm, I've never used json to configure consul, I let spring-cloud do it.

sowston commented 8 years ago

you just drop that json file in the consul config directory. then do consul reload to load the config and the service will be registered in the catalog.

spencergibb commented 8 years ago

I know what it's for, just wondering if that's part of the problem. Hoping to look at this soon.

sowston commented 8 years ago

sure, ok, np. thank u sir!

spencergibb commented 8 years ago

I've had a bit of a look. I didn't use any of your infrastructure setup (too much for me to try and isolate), just the source of your two projects. I ran two instances of RandomNumber and found a problem with the json you gave me. I had to set a unique id for each one, otherwise they stomped on each other. That might be what you are experiencing, only one actually registered with consul. I deployed 2 RandomNumber services, therefore I saw "4 passing" in the consul ui under services. The other issue was you pom.xml mixing releases. I've put an updated one below. Can you try with those changes and see if you are still seeing problems. I also commented out all entries under RandomNumberClient in application.yml` (none of it is needed by default).

randomnumberservice1.json

{
  "service": {
    "id": "RandomNumber1",
    "name": "RandomNumber",
    "tags": ["stateless"],
    "address": "localhost",
    "port": 8081,
    "checks": [
      {
        "id": "RandomNumberRESTGetCheck",
        "http": "http://localhost:8081/services/rest/health",
        "interval": "3s",
        "timeout": "1s"
      }
    ]
  }

}

randomnumberservice0.json

{
  "service": {
    "id": "RandomNumber0",
    "name": "RandomNumber",
    "tags": ["stateless"],
    "address": "127.0.0.1",
    "port": 8080,
    "checks": [
      {
        "id": "RandomNumberRESTGetCheck",
        "http": "http://127.0.0.1:8080/services/rest/health",
        "interval": "3s",
        "timeout": "1s"
      }
    ]
  }
}

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.viasat.services.rest.randomnumber</groupId>
    <artifactId>RandomNumberSpringCloudClient</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>RandomNumberSpringCloudClient</name>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.0.RC1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-starter-parent</artifactId>
                <version>Brixton.M2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <repositories>
        <repository>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>http://repo.spring.io/milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <!--skip deploy -->
                <artifactId>maven-deploy-plugin</artifactId>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!--<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-consul-all</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-feign</artifactId>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <!-- Only needed at compile time -->
            <scope>provided</scope>
        </dependency>
    </dependencies>

</project>

Simplified RandomNumberClient.java

package com.viasat.services.rest.randomnumber;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.feign.EnableFeignClients;
import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import lombok.extern.slf4j.Slf4j;

@Configuration
@EnableAutoConfiguration
@EnableDiscoveryClient
@RestController
@EnableFeignClients
@Slf4j
public class RandomNumberClient {

    @Autowired
    private RandomNumberFeignClient client;

    @RequestMapping("/")
    public String getRandomNumber() {
        return client.getRandomNumber();
    }

    public static void main(String[] args) {
        SpringApplication.run(RandomNumberClient.class, args);
    }

    @FeignClient("RandomNumber")
    public interface RandomNumberFeignClient {
        @RequestMapping(value = "/services/rest/randomnumber", method = RequestMethod.GET)
        String getRandomNumber();
    }
}
sowston commented 8 years ago

Thanks @spencergibb

I'm getting the following error when trying to build with the POM you provided:

Plugin org.springframework.boot:spring-boot-maven-plugin:1.3.0.RC1 or one of its dependencies could not be resolved: Failure to find org.springframework.boot:spring-boot-maven-plugin:jar:1.3.0.RC1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

spencergibb commented 8 years ago

Don't know how that's possible with

    <repositories>
        <repository>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>http://repo.spring.io/milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

http://repo.spring.io/milestone/org/springframework/boot/spring-boot-maven-plugin/1.3.0.RC1/

sowston commented 8 years ago

M2 Eclipse Plugin hell. Great, I no longer see the bahavior after making the ID's unique, simplifying the client and updating the POM. This issue can be closed. Thanks for your help @spencergibb

Totally understand that it's in milestone, but it would be great if the POM's for the examples were up to date and possibly a couple more examples would be extremely helpful.

Nice work sir.

spencergibb commented 8 years ago

NP, we appreciate the feedback!