jankotek / mapdb

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.
https://mapdb.org
Apache License 2.0
4.87k stars 873 forks source link

NPE in ByteBufferVol which appears to corrupt header #1007

Open dan-blum opened 1 year ago

dan-blum commented 1 year ago

We got this exception:

java.lang.NullPointerException at org.mapdb.volume.ByteBufferVol.getSlice(ByteBufferVol.java:42) at org.mapdb.volume.ByteBufferVol.getLong(ByteBufferVol.java:121) at org.mapdb.StoreDirect.longStackTake(StoreDirect.kt:389) at org.mapdb.StoreDirectAbstract.allocateData(StoreDirectAbstract.kt:293) at org.mapdb.StoreDirect.put(StoreDirect.kt:618) at org.mapdb.HTreeMap.valueWrap(HTreeMap.kt:1208) at org.mapdb.HTreeMap.putprotected(HTreeMap.kt:344) at org.mapdb.HTreeMap.put(HTreeMap.kt:324)

This is similar to the exception reported in #963 but not identical.

This appears to have corrupted the MapDB header as upon restart we got

java[11865]: at org.mapdb.DBMaker$Maker.make(DBMaker.kt:450) java[11865]: at org.mapdb.StoreDirect$Companion.make$default(StoreDirect.kt:56) java[11865]: at org.mapdb.StoreDirect$Companion.make(StoreDirect.kt:57) java[11865]: at org.mapdb.StoreDirect.(StoreDirect.kt:114) java[11865]: at org.mapdb.StoreDirectAbstract.fileHeaderCheck(StoreDirectAbstract.kt:113) java[11865]: Exception in thread "main" org.mapdb.DBException$DataCorruption: Header checksum broken. Sto

(This is inverted because it's from a systemctl log.)

We are using MapDB 3.0.8 and it is instantiated like so:

        mapDB = DBMaker
            .fileDB(mapDBFile)
            .closeOnJvmShutdown()
            .fileMmapEnable()
            .fileMmapPreclearDisable()
            .make();

Transactions are not enabled because we need to be able to shrink the file. However, we

  1. Always call commit() after changing the map (there is only one map).
  2. Always change the map inside a Guava Monitor so two threads cannot change it at the same time.