search

grpc-ecosystem / grpc-spring

Spring Boot starter module for gRPC framework.
https://grpc-ecosystem.github.io/grpc-spring/
Apache License 2.0
3.41k stars 808 forks source link

Off-heap memory leak. I'm not sure if it's a grpc-spring problem or a netty problem. #1123

Open hakusai22 opened 5 days ago

hakusai22 commented 5 days ago

Using top -c to check RES memory usage, it was 1.2g on Friday and rose to 1.5g on Monday. Using pmap to check 1539536kb (1.5g), and turning on NMT to check jcmd -p VM.native_memory detail, the total committed memory is 1152483kb (1g). The difference is 377MB. These are all off-heap memories that JVM cannot monitor. The off-heap memory block data at the memory address 00007xxxx has increased, and it is suspected to be an off-heap memory leak. Checking the dump file, it is found that the classes related to off-heap memory are io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue and io.grpc.netty.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue

Below are the memory leak data analyzed by grpc version, netty version and mat tool


<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>1.53.0</version>
</dependency>
    <grpc.spring.version>2.14.0.RELEASE</grpc.spring.version>

        <!-- grpc -->
        <dependency>
            <groupId>net.devh</groupId>
            <artifactId>grpc-client-spring-boot-starter</artifactId>
            <version>${grpc.spring.version}</version>
        </dependency>
        <dependency>
            <groupId>net.devh</groupId>
            <artifactId>grpc-server-spring-boot-starter</artifactId>
            <version>${grpc.spring.version}</version>
        </dependency>


- io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache
<img width="1200" alt="image" src="https://github.com/grpc-ecosystem/grpc-spring/assets/58761705/aa2154ea-1940-48b0-917e-f60a5eb5ee1f">

- io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
<img width="2184" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/68645b9a-bdbc-4cf5-9023-dc3a4181631d">

<img width="1410" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/ce5c11db-4ad3-407b-a943-8899754c293c">

<img width="2173" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/8c6225d3-187e-4205-a869-d0ec79468677">

![img_v3_02ca_9c8c2390-341d-471d-901e-9f1655b3707g](https://github.com/grpc/grpc-java/assets/58761705/ea710d09-6915-48e9-8ba6-bf2a5b67c193)

<img width="1285" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/92eb00f4-0f12-48d9-9676-8620f0423581">
<img width="1400" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/ae0e7a5a-08e4-465a-abf7-6e06f0069d34">
<img width="937" alt="image" src="https://github.com/grpc/grpc-java/assets/58761705/11be81bc-0e46-42aa-add0-f045afe042a0">