DirectByteBuffer leak when using persistent storage

scf37 commented 3 years ago

JDK implementation of FileChannel allocates and caches DirectByteBuffer for every thread. When persistent ehcache is used from large thread pool and serialized cached values are large, JDK keeps similarly sized DirectByteBuffer for every thread

Long explanation

When FileBackedStorageEngine calls channel.read(buffer, position) for heap byte buffer, JDK allocates temporary DirectByteBuffer of the same size and caches it for further usage in thread local cache via sun.nio.ch.Util.getTemporaryDirectBuffer.

If serialized cached values are large (for ex, 10 MB) and cache is called from large thread pool (for ex, 300 threads), FileBackedStorageEngine reads serialized values into heap byte buffers of 10MB in size, forcing JDK to allocate and cache 300 direct buffers of 10 MB each which is 3 GB of unexpected off-heap memory overhead.

Proposed solutions

Use own direct byte buffer pool in FileBackedStorageEngine. This bypasses JDK direct byte buffer cache and allows to control amount of allocated off-heap memory
Read large serialized values in chunks. In this case cached direct byte buffer instances will be of manageable size.

Environment

ehcache-3.6.3 JDK11

Example trace of large allocation

Allocate 3269184
    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:114)
    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
    at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:242)
    at java.base/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:811)
    at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:796)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.readFromChannel(FileBackedStorageEngine.java:437)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.readFromChannel(FileBackedStorageEngine.java:408)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.access$500(FileBackedStorageEngine.java:60)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine$FileChunk.readBuffer(FileBackedStorageEngine.java:594)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine$FileChunk.readValueBuffer(FileBackedStorageEngine.java:576)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.readValueBuffer(FileBackedStorageEngine.java:277)
    at org.terracotta.offheapstore.storage.PortabilityBasedStorageEngine.readValue(PortabilityBasedStorageEngine.java:102)
    at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1954)
    at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
    at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:725)
    at org.ehcache.impl.internal.store.disk.EhcachePersistentConcurrentOffHeapClockCache.compute(EhcachePersistentConcurrentOffHeapClockCache.java:158)
    at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:962)
    at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.put(AbstractOffHeapStore.java:246)
    at org.ehcache.impl.internal.store.tiering.TieredStore.put(TieredStore.java:107)
    at org.ehcache.core.Ehcache.doPut(Ehcache.java:86)
    at org.ehcache.core.EhcacheBase.put(EhcacheBase.java:189)
    at org.ehcache.jsr107.Eh107Cache.put(Eh107Cache.java:175)

Neoclassic commented 2 years ago

You may jdk.nio.maxCachedBufferSize directive to limit the buffer cache size

scf37 commented 2 years ago

@Neoclassic won't work well for high load. It will lead to frequent allocation and deallocation of large native ByteBuffers.

ehcache / ehcache3