search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.37k stars 3.82k forks source link

suspect memory leak in off-heap #8071

Closed lixiangflyin closed 3 years ago

lixiangflyin commented 3 years ago

Hi all, My grpc-java client off-heap memory grows.

gRPC version

java-SDK: java8 server: grpc-java 1.28.0

My application is processing image data. For example, ocr, resize, and other image model processor. Use grpc corresponding to service.

I catch this case. JVM heap is normal. But native memory leaks with time.

USE JVM NMT setting, I see Internal memory grows higher.

Suspect native memory leak.

In addition, USE linux method, SMAPS all memory. I can find many (2-15M) memory.

Size: 14352 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 12312 kB Pss: 12312 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12312 kB Referenced: 12312 kB Anonymous: 12312 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB

Finally, dump suspect memory, find many grpc request and response info(contain image data).

Can you help me? Thanks!

sanjaypujare commented 3 years ago

Couple of comments:

ejona86 commented 3 years ago

15 MB isn't enough for us to obviously point to a memory leak. What reasons do you think there is a memory leak? Was this causing you a particular problem, or did you just notice something that looked wrong?

Netty has a buffer allocator that keeps buffer caches on each thread. You might notice that memory increases just after initial startup as new connections come in. That would be Netty creating new threads until it reaches its maximum pool size (it is a fixed-size pool, but the threads are lazily created), coupled with each thread holding onto a cache. As long as the memory increase stops after a bit, everything would be considered operating as normal.

ejona86 commented 3 years ago

No response to provide more information, so closing. More information can be provided and then we can reopen.