Memory leak in netty4.1 NettyChannelPipelineInstrumentation

[VanillaIceCreams]

Describe the bug

My service had a heap overflow and dump a hrof file，I analyzed the heap memory and found the following problem:

25w unreachable object in reference queue but can not remove them...

Steps to reproduce

The problem happened yesterday and has not been repeated yet.

Expected behavior

not oom

Actual behavior

the server used xxl-job and redission，and they both depend on netty.

Javaagent or library instrumentation version

v1.26

Environment

JDK:java-1.8.0-openjdk-1.8.0.342.b07-0.oe2203.aarch64 OS: [root@rabbitmq1 bin]# uname -a Linux rabbitmq1 5.10.0-60.68.0.93.oe2203.aarch64 #1 SMP Wed Nov 23 14:49:18 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux [root@rabbitmq1 bin]# cat /etc/os-release NAME="openEuler" VERSION="22.03 LTS" ID="openEuler" VERSION_ID="22.03" PRETTY_NAME="openEuler 22.03 LTS" ANSI_COLOR="0;31"

Additional context

No response

[VanillaIceCreams]

[laurit]

Based on just these screenshots it is hard to tell, ideally we'd like to reproduce this. On thing that is a bit strange is that virtual field implementation is using the map fallback, it should be writing the value directly to the object that is used as the key.

[VanillaIceCreams]

Based on just these screenshots it is hard to tell, ideally we'd like to reproduce this. On thing that is a bit strange is that virtual field implementation is using the map fallback, it should be writing the value directly to the object that is used as the key.

I was wrong, but here's what I found: There are additional strong references to HttpServerTracingHandler and chdec！

[VanillaIceCreams]

The number of HttpServerTracingHandler objects is only increasing

[laurit]

@VanillaIceCreams I'd suggest that you first try with the latest 1.x (or 2.x) release and see if you can still reproduce this. v1.26 that you are using is over a year old.

[github-actions[bot]]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.

[will-zdu]

@VanillaIceCreams I think you can try to look at the keys in this map, this key should be referenced elsewhere, from the current screenshot of your mat, I'm not sure if there are no other strong references, this key is handler, from the opentelemetry source code, I haven't seen any memory leaks yet.

[github-actions[bot]]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.

[starsliao]

@laurit @VanillaIceCreams

I also encountered the issue of memory leaks with Netty. I am using version 2.5.0 of autoinstrumentation-java, and I have experienced the same problem.

In long-running Java microservices (running for more than 20 days，high volume of requests), there is insufficient Java heap memory. Many microservices are experiencing this issue, and some of these microservices are not even using Netty.

I previously had the same issue when using version 2.3.0 of autoinstrumentation-java.

This is latest Java dump file.

I am an operations engineer, and this is the phenomenon I observed. Below is the screenshot information provided by my development colleagues.