search

input-output-hk / iodb

Multiversioned key-value database, especially useful for blockchain
Creative Commons Zero v1.0 Universal
95 stars 13 forks source link

StoreShard.taskDistribute fails under heavy load with OOEM #30

Open jankotek opened 6 years ago

jankotek commented 6 years ago

If there are too many inserts, the distribute tasks will run out of heap and fail.

The data merging uses lazy iterator and does not load all data into memory. Problem is in serialization. The entire data set is serialized into on-heap byte[], and that causes OOEM.

Solution is to serialize data into FileChannel and latter transfer it into log file

jankotek commented 6 years ago

Another option is to apply back pressure (slow down inserts), if there are too many undistributed changes.