search

jankotek / mapdb

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.
https://mapdb.org
Apache License 2.0
4.87k stars 873 forks source link

Data corruption when using expireAfterCreate() and expireOverflow() with HashMapMaker #1046

Open igalkin opened 3 months ago

igalkin commented 3 months ago

I am using DBMaker.memoryShardedHashMap() to create a memory map and DBMaker.fileDB() to create a disk map to be used as expireOverflow() for the memory map.

Occasionally (not every time), when data starts getting evicted from memory and put on disk, an attempt to remove entries fails because the storage is corrupted. The stack traces vary.

This seems to be the case only when expireAfterCreate() is set on the disk map, even if not setting any expiration strategy, i.e. no expireStoreSize(), etc. There is no expireExecutor(), hence expiration is synchronous. Would appreciate some insight on the same:

java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 4
    at org.mapdb.DataInput2$ByteArray.unpackInt(DataInput2.java:162)
    at org.mapdb.DataInput2.readUTF(DataInput2.java:491)
    at org.mapdb.serializer.SerializerString.deserialize(SerializerString.java:20)
    at org.mapdb.serializer.SerializerString.deserialize(SerializerString.java:11)
    at org.mapdb.HTreeMap$leafValueExternalSerializer$1.deserialize(HTreeMap.kt:228)
    at org.mapdb.HTreeMap$leafValueExternalSerializer$1.deserialize(HTreeMap.kt:214)
    at org.mapdb.Serializer.deserializeFromLong(Serializer.java:621)
    at org.mapdb.StoreDirect.get(StoreDirect.kt:543)
    at org.mapdb.HTreeMap.expireEvictEntry(HTreeMap.kt:903)
    at org.mapdb.HTreeMap$expireEvictSegment$1.take(HTreeMap.kt:881)
    at org.mapdb.QueueLong.takeUntil(QueueLong.kt:153)
    at org.mapdb.HTreeMap.expireEvictSegment(HTreeMap.kt:859)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:462)
org.mapdb.DBException$GetVoid: Record does not exist, recid=29
    at org.mapdb.StoreDirect.get(StoreDirect.kt:533)
    at org.mapdb.HTreeMap.expireEvictEntry(HTreeMap.kt:903)
    at org.mapdb.HTreeMap$expireEvictSegment$1.take(HTreeMap.kt:881)
    at org.mapdb.QueueLong.takeUntil(QueueLong.kt:153)
    at org.mapdb.HTreeMap.expireEvictSegment(HTreeMap.kt:859)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:462)
org.mapdb.DBException$DataCorruption: nodeRecid not found in this leaf
    at org.mapdb.HTreeMap.expireEvictEntry(HTreeMap.kt:921)
    at org.mapdb.HTreeMap$expireEvictSegment$1.take(HTreeMap.kt:881)
    at org.mapdb.QueueLong.takeUntil(QueueLong.kt:153)
    at org.mapdb.HTreeMap.expireEvictSegment(HTreeMap.kt:859)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:462)
kotlin.KotlinNullPointerException: null
    at org.mapdb.QueueLong.remove(QueueLong.kt:165)
    at org.mapdb.HTreeMap.removeprotected(HTreeMap.kt:493)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:464)
    at org.mapdb.DB$HashMapMaker$verify$listener$1.modify(DB.kt:736)
    at org.mapdb.HTreeMap.listenerNotify(HTreeMap.kt:1189)
    at org.mapdb.HTreeMap.removeprotected(HTreeMap.kt:513)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:464)
org.mapdb.DBException$PointerChecksumBroken: Broken bit parity
    at org.mapdb.DataIO.parity1Get(DataIO.java:440)
    at org.mapdb.DataIO.unpackRecid(DataIO.java:172)
    at org.mapdb.serializer.SerializerRecid.deserialize(SerializerRecid.java:22)
    at org.mapdb.serializer.SerializerRecid.deserialize(SerializerRecid.java:13)
    at org.mapdb.QueueLong$Node$SERIALIZER.deserialize(QueueLong.kt:50)
    at org.mapdb.QueueLong$Node$SERIALIZER.deserialize(QueueLong.kt:39)
    at org.mapdb.StoreDirectAbstract.deserialize(StoreDirectAbstract.kt:231)
    at org.mapdb.StoreDirect.get(StoreDirect.kt:547)
    at org.mapdb.QueueLong.remove(QueueLong.kt:170)
    at org.mapdb.HTreeMap.removeprotected(HTreeMap.kt:493)
    at org.mapdb.HTreeMap.remove(HTreeMap.kt:464)