search

fusesource / leveldbjni

A Java Native Interface to LevelDB
BSD 3-Clause "New" or "Revised" License
536 stars 143 forks source link

LevelDB native memory consumption is increasing #113

Open kaymakyavuz opened 3 years ago

kaymakyavuz commented 3 years ago

We are using leveldbjni-linux64 dependency in our backends which implement only read operations (we are filling the DB beforehand in a separate process). The DB holds about 200mil keys and recieves +300 requests per second. We are seeing an unbounded increase in native memory consumption. Pmap shows +2GB of SST files and +11GB anonymous, when working-set is about 6 GB. I know that under pressure should leveldb give these files back. But that does not happen in our case.

Setting: Cache size: 100MB, max_open_files=default

Krishna7557 commented 3 years ago

@kaymakyavuz, we also face similar issue while running LevelDB on Linux server. When we look at pmap we could see loy anon objects consuming memory. Did you find what is the cause for this?

Regards, Krishna

kaymakyavuz commented 3 years ago

Hi @Krishna7557 , the only solution I found was to tweak the unix env, which I did not test. I have stopped looking for a solution, just massively (x3) scaled deployment. For the time being it is relatively better but not ideal. I am considering alternative options