Leaking PooledMemoryManager$PoolHeapBuffer

[arinban]

I'm stumbling upon OutOfMemoryErrors in my app. Checking one of its heap dumps I see there are 31k byte[] instances worth of ~190MB and 20k instances of PooledMemoryManager$PoolHeapBuffer. When I check with visualvm where those byte[] come from, I see they are indeed related to Grizzly.

I wonder if this is the same bug as #1804 which was closed because it could not be reproduced.

This is how the top 5 heap offenders look like:

num #instances #bytes Class description

1: 31024 192039272 byte[] 2: 66196 5674816 char[] 3: 7596 2495312 int[] 4: 60430 1450320 java.lang.String 5: 19488 1403136 org.glassfish.grizzly.memory.PooledMemoryManager$PoolHeapBuffer ...

LEt me know what extra information is needed to investigate this issue.

Environment

Ubuntu 16.04

[arinban]

• Issue Imported From: https://github.com/javaee/grizzly/issues/1905
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To:@glassfishrobot
• Original Issue Closed By:@glassfishrobot

[arinban]

@glassfishrobot Commented Reported by dwilches

[arinban]

@glassfishrobot Commented dwilches said: I am using Jersey 2.25.1 by the way.

[arinban]

@glassfishrobot Commented @rlubke said: I don't think what you're observing is a leak. The PooledMemoryManager, will allocate a percentage (10% by default) of the JVM's heap across a series of buffers of varied sizes. See [1].

In 2.4.0 (soon to be released), This has been reduced to 3%.

You have a couple of choices, both require implementing the DefaultMemoryManagerFactory [2] interface.

1) Create and return a new PooledMemoryManager tuned to your tastes. See [1] for tuning options. 2) Create and return a HeapMemoryManager instance. This is the memory manager that was used as the runtime default prior to the inclusion of the PooledMemoryManager (it does have slightly worse performance characteristics compared to PooledMemoryManager).

You can then start the JVM with the system property -Dorg.glassfish.grizzly.MEMORY_MANAGER_FACTORY=.

Make sure your implementation includes a public no-arg constructor.

[1] https://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/memory/PooledMemoryManager.html [2] https://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/memory/DefaultMemoryManagerFactory.html

[arinban]

@glassfishrobot Commented @rlubke said: If you can provide a memory dump that shows the instances are being held by an entity other than the PooledMemoryManager, we can look into the issue further, but so far, this hasn't been the case on any of the reports.

[arinban]

@glassfishrobot Commented dwilches said: Hi Ryan, Thanks for your response. I should have mentioned in the issue that I start my JVM with -Xmx900m so I was expecting something like ~90MB of byte[] instances.

I am using the #1 configuration you mentioned there, leaving DEFAULT_HEAP_USAGE_PERCENTAGE as the value for percentOfHeap.

A moment ago, to test the effect of that parameter I just reduced the percentOfHeap to 0.05 and this is how the heap of my freshly restarted app looks like (still doesn't look like the 5%):

num #instances #bytes Class description

1: 34674 150373424 byte[] 2: 146211 12285288 char[] 3: 20786 3568168 int[] 4: 95270 2286480 java.lang.String 5: 42422 1696880 java.util.LinkedHashMap$Entry 6: 18474 1640008 java.util.HashMap$Node[] 7: 25522 1286040 java.lang.Object[] 8: 36228 1159296 java.util.HashMap$Node 9: 14624 1052928 org.glassfish.grizzly.memory.PooledMemoryManager$PoolHeapBuffer

I ran a jstat to see my heap size and I got: NGCMX=307200.0 and OGCMX=614400.0, so heap is really 900MB.

About sending a heap dump, I'd need to make a dummy app for that and try to reproduce the issue first.

[arinban]

@glassfishrobot Commented dwilches said: I took a heap dump of my current app (with the 5% percentOfHeap on the 900MB heap) and I got the following instances that have PooledMemoryManager anywhere in the name:

[arinban]

@glassfishrobot Commented @rlubke said: Can you post the dump to dropbox?

[arinban]

@glassfishrobot Commented dwilches said: The dump could have sensitive information, so I'd need to reproduce it in a dummy app first in order to send it. I'll do it sometime next week.

[arinban]

@glassfishrobot Commented dwilches said: Hello,

I managed to reproduce the problem with a fresh Jersey project:

When I spawned my process I passed -Xmx1g to the JVM, so I expected 100MB worth of byte[], not 200Mb as you see in the image. The heap dump was taken with the following command (note the live parameter, so all those objects are not GarbageCollectable):

jmap -dump:live,format=b,file=heapdump_live.bin 25465

The heap dump and source code are located here: https://drive.google.com/open?id=0B1xs_Kpp-C2oQzljYXo2QlZTN3M Here is also the complete source code:

**Main.java**package com.example;

import org.glassfish.grizzly.http.server.HttpServer;
import org.glassfish.grizzly.http.server.NetworkListener;
import org.glassfish.grizzly.memory.PooledMemoryManager;
import org.glassfish.grizzly.nio.transport.TCPNIOTransport;
import org.glassfish.grizzly.nio.transport.TCPNIOTransportBuilder;

import javax.ws.rs.core.Application;
import java.io.IOException;

public class Main extends Application
{
    private final HttpServer server = new HttpServer();

    private HttpServer startServer()
    {
        NetworkListener networkListener = new NetworkListener("grizzly", "0.0.0.0", 8080);

        // I'm actually passing some other parameters to this constructor in my real app, but this is enough to cause the problem
        PooledMemoryManager pooledMemoryManager = new PooledMemoryManager();
        TCPNIOTransport transport = TCPNIOTransportBuilder.newInstance()
.setMemoryManager(pooledMemoryManager)
.build();
        networkListener.setTransport(transport);
        server.addListener(networkListener);

        return this.server;
    }

    public static void main(String[] args) throws IOException
    {
        final HttpServer server = new Main().startServer();
        System.out.println("Jersey app started");
        System.in.read();
        server.stop();
    }
}

[arinban]

@glassfishrobot Commented @rlubke said: It appears there's a small problem with the test case.

private HttpServer startServer()
    {
        NetworkListener networkListener = new NetworkListener("grizzly", "0.0.0.0", 8080);

        // I'm actually passing some other parameters to this constructor, but this is enough to cause the problem
        PooledMemoryManager pooledMemoryManager = new PooledMemoryManager();
        TCPNIOTransport transport = TCPNIOTransportBuilder.newInstance()
.setMemoryManager(pooledMemoryManager)
.build();
        networkListener.setTransport(transport);
        server.addListener(networkListener);

        return this.server;
    }

By instantiating the PooledMemoryManager in this fashion, you're effectively created a second PooledMemoryManager instances in parallel to the one associated with the MemoryManager.DEFAULT_MEMORY_MANAGER instance. Hence, twice the memory.

As I mentioned, and maybe I didn't explain well enough, is that if you want to impact the overall VM footprint, you will need to provide an implementation of the DefaultMemoryManagerFactory:

/**
 * Allows creation/customization of {@link MemoryManager} implementation to
 * be used as the {@link MemoryManager#DEFAULT_MEMORY_MANAGER}.
 *
 * Specify a custom <code>DefaultMemoryManagerFactory</code> implementation using
 * a JVM system property {@value #DMMF_PROP_NAME}.  Implementations of this interface
 * will be created by invoking its no-arg constructor.
 */
public interface DefaultMemoryManagerFactory {

    String DMMF_PROP_NAME = "org.glassfish.grizzly.MEMORY_MANAGER_FACTORY";

    /**
     * @return the MemoryManager that should be used as the value of
     * {@link MemoryManager#DEFAULT_MEMORY_MANAGER}.
     * This method should return <code>null</code> if the {@link MemoryManager}
     * could not be created.
     */
    MemoryManager createMemoryManager();

}

Within your createMemoryManager implementation, you would create a PooledMemoryManager tuned to your desired settings. The value returned here will be set as the value of MemoryManager.DEFAULT_MEMORY_MANAGER and will impact the footprint of the VM in the fashion you're expecting.

NOTE: In order for the factory to be used, per the docs, you need to provide the class reference as a property to the JVM: -Dorg.glassfish.grizzly.MEMORY_MANAGER_FACTORY=

[arinban]

@glassfishrobot Commented dwilches said: Indeed, using a custom MemoryManagerFactory solved our problems. Now the byte[] instances are using the expected amount memory. Thank you very much.

[arinban]

@glassfishrobot Commented This issue was imported from java.net JIRA GRIZZLY-1905

[arinban]

@glassfishrobot Commented Marked as works as designed on Monday, April 24th 2017, 6:29:10 pm