@AuthorizedFeignClient invoke other services throw timeout

anjia0532 commented 7 years ago

jhipster-registry v2.5.7
uaa v3.12.2
microservices1(e.g. named app1) v3.12.2
microservices2(e.g. named app2) v3.12.2

when the first time app1 invoked the app2 thow time-out exception,but since the second is successful.

Exception in com.shunneng.app1.web.rest.App1Resource.getApp1s() with cause = 'java.util.concurrent.TimeoutException' and exception = 'invokeWebService timed-out and no fallback available.'

com.netflix.hystrix.exception.HystrixRuntimeException: invokeWebService timed-out and no fallback available.
    at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:783)
    at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:768)
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1448)
    at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1373)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at rx.observers.Subscribers$5.onError(Subscribers.java:230)
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10211)
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:142)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:1099)
    at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
    at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
    at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
    at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:1116)
    at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: null
    at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:958)
    at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:59)
    at com.netflix.hystrix.AbstractCommand$11.call(AbstractCommand.java:573)
    at com.netflix.hystrix.AbstractCommand$11.call(AbstractCommand.java:565)
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
    ... 15 common frames omitted

App2FeignClient

@AuthorizedFeignClient(name="app2")
public interface App2FeignClient {
    @RequestMapping(value = "/api/app2s",
        method = RequestMethod.GET,
        produces = MediaType.APPLICATION_JSON_VALUE)
    ResponseEntity<List<App2DTO>> getApp2s();
}

AuthorizedFeignClient


import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.cloud.netflix.feign.FeignClientsConfiguration;
import org.springframework.core.annotation.AliasFor;

import java.lang.annotation.*;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@FeignClient
public @interface AuthorizedFeignClient {

    @AliasFor(annotation = FeignClient.class, attribute = "name")
    String name() default "";

    /**
     * A custom <code>@Configuration</code> for the feign client.
     *
     * Can contain override <code>@Bean</code> definition for the pieces that
     * make up the client, for instance {@link feign.codec.Decoder},
     * {@link feign.codec.Encoder}, {@link feign.Contract}.
     *
     * @see FeignClientsConfiguration for the defaults
     */
    @AliasFor(annotation = FeignClient.class, attribute = "configuration")
    Class<?>[] configuration() default OAuth2InterceptedFeignConfiguration.class;

    /**
     * An absolute URL or resolvable hostname (the protocol is optional).
     */
    String url() default "";

    /**
     * Whether 404s should be decoded instead of throwing FeignExceptions.
     */
    boolean decode404() default false;

    /**
     * Fallback class for the specified Feign client interface. The fallback class must
     * implement the interface annotated by this annotation and be a valid Spring bean.
     */
    Class<?> fallback() default void.class;

    /**
     * Path prefix to be used by all method-level mappings. Can be used with or without
     * <code>@RibbonClient</code>.
     */
    String path() default "";
}

Welcome to the JHipster Information Sub-Generator

JHipster Version(s)

app1
`-- (empty)

JHipster configuration, a `.yo-rc.json` file generated in the root folder

{
  "generator-jhipster": {
    "jhipsterVersion": "3.12.2",
    "baseName": "app1",
    "packageName": "com.shunneng.app1",
    "packageFolder": "com/shunneng/app1",
    "serverPort": "8081",
    "authenticationType": "uaa",
    "uaaBaseName": "uaa",
    "hibernateCache": "hazelcast",
    "clusteredHttpSession": false,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "mysql",
    "searchEngine": false,
    "messageBroker": false,
    "serviceDiscoveryType": "eureka",
    "buildTool": "maven",
    "enableSocialSignIn": false,
    "jwtSecretKey": "f690b1c9a3913b7132beec778a626a17930e576d",
    "enableTranslation": true,
    "applicationType": "microservice",
    "testFrameworks": [
      "gatling"
    ],
    "jhiPrefix": "jhi",
    "skipClient": true,
    "skipUserManagement": true,
    "nativeLanguage": "zh-cn",
    "languages": [
      "zh-cn"
    ]
  }
}

Entity configuration(s) `entityName.json` files generated in the `.jhipster` directory

Appointment.json

{
    "fluentMethods": true,
    "relationships": [],
    "fields": [
        .....
    ],
    "changelogDate": "20161226075125",
    "javadoc": "app1",
    "entityTableName": "app1",
    "dto": "no",
    "pagination": "no",
    "service": "no"
}

Browsers and Operating System

java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

git version 2.10.1.windows.1

node: v7.2.1

npm: 3.10.10

bower: 1.8.0

gulp: [11:08:41] CLI version 1.2.2

yeoman: 1.8.5

Docker version 1.12.2-rc1, build 45bed2c, experimental

docker-compose version 1.8.1, build 004ddae

pascalgrimaud commented 7 years ago

Can you try to put this configuration ?

zuul:
    host:
        connect-timeout-millis: 5000
        socket-timeout-millis: 10000
# See https://github.com/Netflix/Hystrix/wiki/Configuration
hystrix:
    command:
        default:
            execution:
                isolation:
                    thread:
                        timeoutInMilliseconds: 10000

Currently, this config is only for gateway. I think we should extend it to microservices

anjia0532 commented 7 years ago

yes, this is my gateway\src\main\resources\config\application.yml

ribbon:
    ReadTimeout: 60000
    eureka:
        enabled: true
zuul:
    host:
        connect-timeout-millis: 5000
        socket-timeout-millis: 60000

# See https://github.com/Netflix/Hystrix/wiki/Configuration
hystrix:
    command:
        default:
            execution:
                timeout:
                    enabled: false
                isolation:
                    strategy: THREAD
                    thread:
                        timeoutInMilliseconds: 60000

and it's not real time-out,because the since the second is successful,the first invoke must be failure.

pascalgrimaud commented 7 years ago

I already have the same issue like you: 1st request failed with timeout, 2nd successful You have to put this configuration in your microservices, not only in gateway

anjia0532 commented 7 years ago

ok,i'll have a try. thank you :)

gmarziou commented 7 years ago

Reason why first request on a proxied service takes longer and may timeout is because ribbon clients are lazy instantiated.

See https://github.com/spring-cloud/spring-cloud-netflix/issues/1334 for a workaround.

xetys commented 7 years ago

hey,

i currently running this configuration in JHipster Registry for overall configuration:

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 20000
ribbon:
  MaxAutoRetries: 3
  ReadTimeout: 5000
  eureka:
    enabled: true

which you then can override using specialized config files, as well as env vars etc.

with this, my feign clients do also retries. I hope this configs are still the right one, if nothing changed since last cloud releases

pascalgrimaud commented 7 years ago

I'm closing this, as I'm pretty sure these configuration can be a solution.

@xetys : there are a lot of different configurations, but it depends on business case

xetys commented 7 years ago

Yes I know. I added my to show the ribbon configuration, which was missing in the examples before

jhipster / generator-jhipster