Netflix / Hystrix

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
24.07k stars 4.7k forks source link

Too many invocation of method proteced by HystrixCommand #1682

Open Clueless76 opened 6 years ago

Clueless76 commented 6 years ago

Hello,

since we activated HystricCommand our code ends up in a mass.

We have a bean in Wildfly 10 with following declaration:

@Stateless
@Remote
public class Bean implements BeanInterface {
    @Inject
    private Adapter adapter;

    @Override
    public Result getDebitData(Request request) {
        return adapter.getDebitData(request);
    }
}

The adapter was protected by the HystrixCommand:

public class Adapter {
    @Inject
    private RestConnector connector;

        @HystrixCommand
    @Interceptors({ HystrixInterceptor.class })
    public Result getDebitData(Request request) {
        Response clientResponse = connector.connect().getDebitData(request);
        checkHttpStatusCodeWasSuccess(clientResponse);
        return clientResponse.readEntity(Result .class);
    }
}

Without the HystrixCommand annotation the code runs correctly. Since we activated the command we see that immideately that the command is exectued many times:

2017-09-27 15:32:41,971 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-6) execute command: getDebitData
2017-09-27 15:32:41,971 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-9) execute command: getDebitData
2017-09-27 15:32:41,972 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-1) execute command: getDebitData
2017-09-27 15:32:41,974 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-8) execute command: getDebitData
2017-09-27 15:32:41,974 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-10) execute command: getDebitData
2017-09-27 15:32:41,974 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-2) execute command: getDebitData
2017-09-27 15:32:41,975 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-3) execute command: getDebitData
2017-09-27 15:32:41,975 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-5) execute command: getDebitData
2017-09-27 15:32:41,975 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-7) execute command: getDebitData
2017-09-27 15:32:41,976 DEBUG [com.netflix.hystrix.contrib.javanica.command.GenericCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-4) execute command: getDebitData
2017-09-27 15:32:41,976 DEBUG [com.netflix.hystrix.AbstractCommand] (hystrix-Adapter$Proxy$_$$_WeldSubclass-4) HystrixCommand execution REJECTED_THREAD_EXECUTION and fallback failed.: java.lan
g.RuntimeException: No fallback available.
[...]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ad88423 rejected from java.util.concurrent.ThreadPoolExecutor@797ba7e8[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 180]

The code is now in a bad situation because the pool is fuel. Exception are thrown. The code of the adpater is never executed.

Do you have an idea what happened? How can I get a correct behaviour?

In System properties of Wildfly the only property which is set is: hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds 60000

Thank you for your help. Markus

Clueless76 commented 6 years ago

Hi,

I investigated now further this issue and find out if I remove the "@HysterixCommand" from the adapter class and add it to the bean it works.

So far as I understand there is a use of CDI and WELD but I'm not able to solve it as expected.

Cheers, Markus