search

michaelhyatt / elastic-apm-mule3-agent

Elastic APM agent for Mule 3.x
Apache License 2.0
19 stars 7 forks source link

APM spanstore is hogging heap memory #42

Closed jpandher1991 closed 2 years ago

jpandher1991 commented 3 years ago

Hi

we are running APM domain tracer 1.17.0 with mule 3.9.4.And we have close to 20 apps running under one domain and in JVM. Somehow we see our heap memory is getting hogged by co.elastic.apm.mule.utils.SpanStore class .I have attached the screenshot with the ticket.Also this heap dump is for one particular time when it is only consuming 899MB out of 1.9GB. memory hogging keep on growing over time.Can you please suggest something

Screen Shot 2021-02-03 at 6 40 20 PM
michaelhyatt commented 3 years ago

I suspect that transactions entries in the HashMap at the core of the SpanStore class are not being garbage collected. Will have a look at it.

michaelhyatt commented 3 years ago

Upgraded my Mac to Big Sur only to find out that Anypoint 6.6 is not compatible with it:

https://docs.mulesoft.com/release-notes/studio/anypoint-studio-6.6-with-3.9.4-runtime-update-site-7-release-notes

Found the workaround to install it on a virtual machine, but this seems to require a bit more time to get it resolved. So, stay tuned.

michaelhyatt commented 3 years ago

@jpandher1991 I did a bit of a change to handle memory allocations a bit better, haven't seen any memory issues in my environment. Do you mind giving it a spin and testing it as well in your environment before I merge it and create a release ? I am attaching the new jar to this issue.

apm-mule3-agent-1.21.0.zip

jpandher1991 commented 3 years ago

Thanks michael, I will test it out and let you know

jpandher1991 commented 3 years ago

Hi Michael

So we tried the new jar. Still the same issue. Heap memory is getting hogged by concurrent hash map.

michaelhyatt commented 2 years ago

Fixed in v1.21.1