search

Netflix / Hystrix

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
24.07k stars 4.7k forks source link

High CPU% in rx.internal.operators.OperatorScan$3.onNext(OperatorScan.java:138) #1703

Open RajibBiswasTibco opened 6 years ago

RajibBiswasTibco commented 6 years ago

Hi, I am using Hystrix 1.5.3 and facing following issue: When I try to run some application in Pivotal Cloud Foundry (PCF), and when the application is bind with Hystrix dashboard service in PCF, we are observing high CPU usage. I have attached the thread dumps related to this issue. high_td.log td1.log

Timer-0 thread is consuming most of the CPU as can be seen from the thread dumps. In td1.log, you can see following stack trace:

"Timer-0" #99 daemon prio=5 os_prio=0 tid=0x00007f47b568b800 nid=0x117 runnable [0x00007f47a5c46000] java.lang.Thread.State: RUNNABLE at rx.internal.operators.OperatorScan$3.onNext(OperatorScan.java:138) at rx.internal.operators.UnicastSubject$State.replay(UnicastSubject.java:262) at rx.internal.operators.UnicastSubject$State.onNext(UnicastSubject.java:149) at rx.internal.operators.UnicastSubject.onNext(UnicastSubject.java:68) at rx.internal.operators.OperatorWindowWithSize$InexactSubscriber.onNext(OperatorWindowWithSize.java:207) at rx.observers.Subscribers$5.onNext(Subscribers.java:229) at rx.observers.SerializedObserver.onNext(SerializedObserver.java:95) at rx.observers.SerializedSubscriber.onNext(SerializedSubscriber.java:95) at rx.internal.operators.OperatorConcat$ConcatInnerSubscriber.onNext(OperatorConcat.java:214) at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.fastpath(OnSubscribeFromIterable.java:127) at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:70) at rx.internal.producers.ProducerArbiter.setProducer(ProducerArbiter.java:126) at rx.internal.operators.OperatorConcat$ConcatInnerSubscriber.setProducer(OperatorConcat.java:238) at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:49) at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:32) at rx.Observable.unsafeSubscribe(Observable.java:8314) at rx.internal.operators.OperatorConcat$ConcatSubscriber.subscribeNext(OperatorConcat.java:176) at rx.internal.operators.OperatorConcat$ConcatSubscriber.onNext(OperatorConcat.java:138) at rx.internal.operators.OperatorConcat$ConcatSubscriber.onNext(OperatorConcat.java:75) at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.slowPath(OnSubscribeFromArray.java:100) at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:63) at rx.Subscriber.setProducer(Subscriber.java:211) at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32) at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24) at rx.Observable$2.call(Observable.java:162) at rx.Observable$2.call(Observable.java:154) at rx.Observable.unsafeSubscribe(Observable.java:8314) at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) at rx.Observable$2.call(Observable.java:162) at rx.Observable$2.call(Observable.java:154) at rx.Observable$2.call(Observable.java:162) at rx.Observable$2.call(Observable.java:154) at rx.Observable$2.call(Observable.java:162) at rx.Observable$2.call(Observable.java:154) at rx.Observable.subscribe(Observable.java:8407) at rx.Observable.subscribe(Observable.java:8374) at rx.Observable.subscribe(Observable.java:8271) at com.netflix.hystrix.metric.consumer.BucketedCounterStream.startCachingStreamValuesIfUnstarted(BucketedCounterStream.java:88) at com.netflix.hystrix.metric.consumer.BucketedCounterStream.getLatest(BucketedCounterStream.java:103) at com.netflix.hystrix.metric.consumer.RollingCommandEventCounterStream.getLatest(RollingCommandEventCounterStream.java:95) at com.netflix.hystrix.HystrixCommandMetrics.getRollingCount(HystrixCommandMetrics.java:251) at com.netflix.hystrix.HystrixCommandMetrics.getRollingCount(HystrixCommandMetrics.java:265) at com.tibco.bw.netflix.hystrix.dashboard.CloudFoundryHystrixMetricsPublisher.gatherMetrics(CloudFoundryHystrixMetricsPublisher.java:240) at com.tibco.bw.netflix.hystrix.dashboard.CloudFoundryHystrixMetricsPublisher$1.run(CloudFoundryHystrixMetricsPublisher.java) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)

Do you need anything else from my side to look into this issue?

We are using following properties: execution.isolation.strategy = THREAD execution.isolation.thread.timeoutInMilliseconds = 1000 execution.timeout.enabled = true circuitBreaker.requestVolumeThreshold = 300 circuitBreaker.errorThresholdPercentage = 50 circuitBreaker.sleepWindowInMilliseconds = 5000 metrics.rollingStats.timeInMilliseconds = 120000 metrics.rollingStats.numBuckets = 120 requestLog.enabled = true coreSize = 40 maxQueueSize = -1 queueSizeRejectionThreshold = 100 keepAliveTimeMinutes = 1

RajibBiswasTibco commented 6 years ago

Any update on this one?

zopechetan commented 6 years ago

I am also facing same issue , can we have an update ?