search

input-output-hk / iodb

Multiversioned key-value database, especially useful for blockchain
Creative Commons Zero v1.0 Universal
95 stars 13 forks source link

Background exceptions #29

Closed jankotek closed 6 years ago

jankotek commented 6 years ago

Right now the background tasks are failing with some exceptions:

OEM

this is related to issue #28 

SEVERE: Background task failed
java.lang.OutOfMemoryError: Java heap space
        at io.iohk.iodb.LogStore.fileReadData(LogStore.scala:1109)
        at io.iohk.iodb.LogStore.$anonfun$fileReadKeyValues$1(LogStore.scala:1165)
        at io.iohk.iodb.LogStore.$anonfun$fileReadKeyValues$1$adapted(LogStore.scala:1163)
        at io.iohk.iodb.LogStore$$Lambda$1598/623407412.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$Lambda$106/2134156020.apply(Unknown Source)
        at scala.collection.immutable.Range.foreach(Range.scala:156)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at io.iohk.iodb.LogStore.fileReadKeyValues(LogStore.scala:1163)
        at io.iohk.iodb.LogStore.$anonfun$loadKeyValues$1(LogStore.scala:758)
        at io.iohk.iodb.LogStore$$Lambda$1618/193971037.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$Lambda$106/2134156020.apply(Unknown Source)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at io.iohk.iodb.LogStore.loadKeyValues(LogStore.scala:758)
        at io.iohk.iodb.ShardedStore.taskDistribute(ShardedStore.scala:154)
        at io.iohk.iodb.ShardedStore.$anonfun$update$1(ShardedStore.scala:88)
        at io.iohk.iodb.ShardedStore$$Lambda$1606/164611973.apply$mcV$sp(Unknown Source)
        at io.iohk.iodb.Store.$anonfun$runnable$1(Store.scala:184)
        at io.iohk.iodb.Store$$Lambda$1607/1406013867.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

file close race condition

ShardedStore#close() only blocks writers in other threads. If there is reader in other thread (most likely compaction or other background task), it fails with.

SEVERE: Background task failed
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
        at io.iohk.iodb.Utils.readFully(Utils.java:213)
        at io.iohk.iodb.LogStore.fileReadData(LogStore.scala:1111)
        at io.iohk.iodb.LogStore.$anonfun$fileReadKeyValues$1(LogStore.scala:1165)
        at io.iohk.iodb.LogStore.$anonfun$fileReadKeyValues$1$adapted(LogStore.scala:1163)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:156)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at io.iohk.iodb.LogStore.fileReadKeyValues(LogStore.scala:1163)
        at io.iohk.iodb.LogStore.$anonfun$loadKeyValues$1(LogStore.scala:758)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at io.iohk.iodb.LogStore.loadKeyValues(LogStore.scala:758)
        at io.iohk.iodb.ShardedStore.taskDistribute(ShardedStore.scala:154)
        at io.iohk.iodb.ShardedStore.$anonfun$update$1(ShardedStore.scala:88)
        at io.iohk.iodb.Store.$anonfun$runnable$1(Store.scala:184)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
jankotek commented 6 years ago

That is now fixed, I reworked background task scheduling