Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
23.98k
stars
4.7k
forks
source link
Observed latency while using hystrix's queue() method for async tasks #1967
I have a web service and till now the request threads would do async work (like populating cache in case of a cache miss) by using an ExecutorService. But this would not allow timeouts so I decided to use hystrix.
For async tasks it is advised to use HystrixCommand's queue() method. But including hystrix in my service increased latencies of the get requests. So I had to remove hystrix, but since I needed to include timeouts for the async tasks I used both, first ExecutorService's submit() and in that thread wrapped the callables in Hystrix which got the latencies back to normal.
So to summarise my question is, can Hystrix make service calls latent? And how to solve this, as my current solution seems a bit of an hack.
On reading a bit, I learned about Hystrix using a BlockingQueue, can the queue being full and threads waiting to submit a task be the reason for this?
Some info - the number of async tasks we are talking about here is on average around 5k/s and can be as high as 20k/s on a single box.
I have a web service and till now the request threads would do async work (like populating cache in case of a cache miss) by using an ExecutorService. But this would not allow timeouts so I decided to use hystrix.
For async tasks it is advised to use HystrixCommand's queue() method. But including hystrix in my service increased latencies of the get requests. So I had to remove hystrix, but since I needed to include timeouts for the async tasks I used both, first ExecutorService's submit() and in that thread wrapped the callables in Hystrix which got the latencies back to normal.
So to summarise my question is, can Hystrix make service calls latent? And how to solve this, as my current solution seems a bit of an hack.
On reading a bit, I learned about Hystrix using a BlockingQueue, can the queue being full and threads waiting to submit a task be the reason for this?
Some info - the number of async tasks we are talking about here is on average around 5k/s and can be as high as 20k/s on a single box.