search

steveyen / gkvlite

Simple, ordered, key-value persistence library for the Go Language
MIT License
264 stars 27 forks source link

Memory usage during CopyTo #16

Closed cbehopkins closed 6 years ago

cbehopkins commented 7 years ago

I have a large database I am trying to compact. For this I am using CopyTo as described in the readme. The problem is it blows the memory on the machine I am running it on (RaspberryPi)*

Some digging later and I believe the issue is in func (o *Store) visitNodes It recursively calls itself and while doing so keeps the pointers choiceT and choiceF both valid. This seems to mean that every Itemloc structure that the visitNodes traverses during the compact phase has to be held in memory until the end of this recursive traversal.

I have experimented with some fixes that reduce the memory required, but fundamentally this recursive process limits the size of the database that can be traversed to some function of available system memory.

I've attached a memory profile diagram of the program captured shortly before it crashed due to lack of memory.

*yes I know I could in theory run this on a machine with more memory - but it's an easy way to show the scalability issue in question here. mem

steveyen commented 7 years ago

Thanks for the report and profile! Indeed -- it appears visitNodes() will load all the nodes into memory as it proceeds along. Hmmm. And, I don't have any simple one-liner fix that pops to mind.

Related, gkvlite also hasn't been loaded in my mind for awhile now, as most of my efforts related to a golang storage engine have moved to the github.com/couchbase/moss project as it's better aligned to my current needs.

cbehopkins commented 7 years ago

So I'm currently looking at a multi-line fix myself. Struggling to understand how some of this is supposed to work, but will report back if I make any progress. If you get a Pull request you will know I've had some success.

Thanks

Chris

On 10 May 2017 at 16:30, Steve Yen notifications@github.com wrote:

Thanks for the report and profile! Indeed -- it appears visitNodes() will load all the nodes into memory as it proceeds along. Hmmm. And, I don't have any simple one-liner fix that pops to mind.

Related, gkvlite also hasn't been loaded in my mind for awhile now, as most of my efforts related to a golang storage engine have moved to the github.com/couchbase/moss project as it's better aligned to my current needs.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/steveyen/gkvlite/issues/16#issuecomment-300519990, or mute the thread https://github.com/notifications/unsubscribe-auth/AQ6OU6iUMR8NEKsyWD0KVE_q6LkMSTc4ks5r4dgRgaJpZM4NWhua .

cbehopkins commented 6 years ago

Forgot to say I fixed this with 7263ff800f532914be241df8a539e71088c853dc I'm working on a bunch of other fixes and in particular trying to make it golint clean. Will produce a pull request when this process is finished