high disk and memory usage

patrickfrey / strusWebService

web service (HTTP/JSON) to use the strus API as a service

http://project-strus.net

Mozilla Public License 2.0

4 stars 0 forks source link

high disk and memory usage #64

Open andreasbaumann opened 6 years ago

andreasbaumann commented 6 years ago

Running updates for several months without stopping/starting (respectively using the close index command) results in high disk usage (in our project 700 MB to 36 GB). This naturally also increases the amount of memory needed to cache the first two levels of the leveldb structures.

andreasbaumann commented 6 years ago

This bug is critical, it renders the strusWebService not ready for productive environments!

andreasbaumann commented 6 years ago

issuing a /strus/index/close/xxx doens't solve the problem. Only restarting the strusWebSerivce.

andreasbaumann commented 6 years ago

This bug also happens when doing bigger inserts (like 100'000 documents). It seems calling commit doens't do anything on the LevelDB layer to properly organize the data structure.