jankotek / mapdb

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.
https://mapdb.org
Apache License 2.0
4.89k stars 873 forks source link

Out of Direct buffer memory #903

Closed bianjft closed 6 years ago

bianjft commented 6 years ago

The crash when firest full gc because: Out of Direct buffer memory!

exception:

Exception in thread "main" org.mapdb.DBException$OutOfMemory: Out of Direct buffer memory. Increase it with JVM option '-XX:MaxDirectMemorySize=10G'
    at org.mapdb.volume.ByteBufferMemoryVol.ensureAvailable(ByteBufferMemoryVol.java:112)
    at org.mapdb.StoreDirect.allocateNewPage(StoreDirect.kt:153)
    at org.mapdb.StoreDirectAbstract.allocateData(StoreDirectAbstract.kt:307)
    at org.mapdb.StoreDirect.longStackNewChunk(StoreDirect.kt:368)
    at org.mapdb.StoreDirect.longStackPut(StoreDirect.kt:288)
    at org.mapdb.StoreDirectAbstract.releaseData(StoreDirectAbstract.kt:364)
    at org.mapdb.StoreDirect.delete(StoreDirect.kt:735)
    at org.mapdb.HTreeMap.removeprotected(HTreeMap.kt:511)
    at org.mapdb.HTreeMap.expireEvictEntry(HTreeMap.kt:914)
    at org.mapdb.HTreeMap$expireEvictSegment$1.take(HTreeMap.kt:881)
    at org.mapdb.QueueLong.takeUntil(QueueLong.kt:153)
    at org.mapdb.HTreeMap.expireEvictSegment(HTreeMap.kt:859)
    at org.mapdb.HTreeMap.put(HTreeMap.kt:322)
    at javasource.Test.main(Test.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:693)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at org.mapdb.volume.ByteBufferMemoryVol.ensureAvailable(ByteBufferMemoryVol.java:103)
    ... 18 more

set vm options -XX:MaxDirectMemorySize=1G

test code :

public class Test {

    public static void main(String... args){
        DB db = DBMaker.memoryDirectDB().cleanerHackEnable().executorEnable().concurrencyScale(16)
                .make();

        final HTreeMap<Long, byte[]> map = db.hashMap("name_of_map")
                .keySerializer(Serializer.LONG)
                .valueSerializer(Serializer.BYTE_ARRAY)
                .expireAfterCreate(5, TimeUnit.SECONDS)
                .expireMaxSize(1000)
                .expireStoreSize(1024*1024)
                .create();

        final byte[] val = buildBody();

        while(true) {
            long num = 0l;
            while (num < 25 * 1024 * 20) {
                num++;
                map.put(num, val);
            }
        }
    }

    public static byte[] buildBody(){
        String body = "";
        try {
            for (int ii = 0; ii < 1000; ii++) {
                body = body + "body";
            }
            return body.getBytes();
        }finally {
            System.out.println("body size:"+ body.getBytes().length);
        }
    }
}
bianjft commented 6 years ago

pom set mapdb version:3.0.5 and 3.0.7

bianjft commented 6 years ago

i think i solve this problem use 'expireCompactThreshold(0.5)' in set HTreeMap params

   final HTreeMap<String, byte[]> map = db.hashMap("MapDBTest")
                .keySerializer(Serializer.STRING)
                .valueSerializer(Serializer.BYTE_ARRAY)
                .expireAfterCreate(1, TimeUnit.MINUTES)
                .expireMaxSize(1000)
                .expireStoreSize(1024)
                .expireCompactThreshold(0.5)
                .create();