search

Netflix / Hystrix

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
24.1k stars 4.7k forks source link

The DynamicIntegerProperty will case heap overflow and memeory leaks #1735

Open evvil opened 6 years ago

evvil commented 6 years ago

@benjchristensen @spencergibb Hi, when i use the hystrix to dynamically update the config of the hysitrx, it cause heap overflow. I use jmap to dump the heap, and use MAT to analyze the heap.log, I found there were lots of big objects which causes memory leaks.

image

As we can see, it was caused by adding the callback() continuously. image

After analyzing, I found it has differences from DynamicIntegerProperty and DynamicIntProperty. We can find

1. archaius-core-0.6.0-sources.jar!\com\netflix\config\PropertyWrapper.java

image

2. com.netflix.hystrix.strategy.properties.HystrixPropertiesChainedArchaiusProperty.DynamicIntegerProperty

image

And if it can't find the cache in the private static final IdentityHashMap<Class<? extends PropertyWrapper>, Object> SUBCLASSES_WITH_NO_CALLBACK , it will add the callback,

So, does it spell wrong to cause the memory leaks?

yanglifan commented 6 years ago

Still have this issue? You'd better describe more details about the environment and steps to produce this issue. Like which version Hystrix did you use.

evvil commented 6 years ago

sorry, i didn't notice it during Srping Festival @yanglifan I think this problem sitll has, becaues the inappropriate use of our behaviors, we rebuilded lots of hystrix's thread pool during using for hot reloading. And the hystrix use dynamic property to store these properties. When we rebuild these tread pools, we didn't remove these properties and caused these problems.

ashish4710 commented 5 years ago

Is there any new version of this library coming up with this fix?

vipinjnair commented 3 years ago

I am also facing similar memory consumption issue with Hystrix Dynamic properties. @evvil Could you tell more about how you resolved the issue. "When we rebuild these tread pools, we didn't remove these properties and caused these problems." Which properties you are taking about and how you removed them ?

evvil commented 2 years ago

we download the source, and change the code in the source, and push jar to our company's maven repo.

vipinjnair @.***> 于2021年7月15日周四 16:07写道：

I am also facing similar memory consumption issue with Hystrix Dynamic properties. @evvil https://github.com/evvil Could you tell more about how you resolved the issue. "When we rebuild these tread pools, we didn't remove these properties and caused these problems." Which properties you are taking about and how you removed them ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Netflix/Hystrix/issues/1735#issuecomment-880488078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5VDPEXIR33YBFHUVHTCCDTX2JK5ANCNFSM4EIHGULA .