Limiting memory use

[jimrobinson]

Hi,

I am doing something wrong when I use gkvlite in this example, but I'm not sure what. I thought that periodic calls to the Collection.Write and Store.Flush would limit the in-memory size of the program, but that doesn't seem to be the case.

Could I ask you what I'm doing wrong in the following self contained example?

http://play.golang.org/p/RHmZWRIDet

An example of the memory usage as the number of keys inserted is increased:

$ for n in 100 200 400 800 1600 3200 6400 128000 256000 512000; do /usr/bin/time -f '%C %MkB' ./grow -n ${n}; done 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile047548036/mem.pprof ./grow -n 100 6752kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile335212671/mem.pprof ./grow -n 200 6928kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile506557173/mem.pprof ./grow -n 400 7424kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile956633517/mem.pprof ./grow -n 800 7840kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile050576843/mem.pprof ./grow -n 1600 9824kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile875078316/mem.pprof ./grow -n 3200 11776kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile891669337/mem.pprof ./grow -n 6400 15648kB 2014/01/22 18:51:18 profile: memory profiling enabled, /tmp/profile244818683/mem.pprof ./grow -n 128000 159392kB 2014/01/22 18:51:24 profile: memory profiling enabled, /tmp/profile906229215/mem.pprof ./grow -n 256000 280112kB 2014/01/22 18:51:37 profile: memory profiling enabled, /tmp/profile813873422/mem.pprof ./grow -n 512000 580768kB

The memory profile appears to indicate the memory is all in gkvlite:

Adjusting heap profiles for 1-in-4096 sampling rate Welcome to pprof! For help, type 'help'. (pprof) top10 Total: 16.0 MB 7.8 48.7% 48.7% 7.8 48.7% github.com/steveyen/gkvlite.(_Collection).mkNode 5.0 31.6% 80.3% 12.8 80.3% github.com/steveyen/gkvlite.(_Collection).Set 1.2 7.8% 88.1% 1.2 7.8% github.com/steveyen/gkvlite.(_itemLoc).write 1.2 7.8% 95.8% 1.2 7.8% github.com/steveyen/gkvlite.(_nodeLoc).write 0.7 4.2% 100.0% 15.8 99.3% main.main

[d2g]

Doesn't seem to be anything wrong with your example. I'm witnessing the same issue, so suspect that it's a genuine defect.

[jimrobinson]

Thank you for the feedback. I've also tried inserting a call to runtime.GC() within the loop, to see whether or not the issue was just that the collector wasn't getting a chance to run, but I did not see any difference in behavior.

[steveyen]

Looks like you're inserting new items at a rate 1000 times more than calls to EvictSomeItems().

You can see EvictSomeItems() is pretty simple...

https://github.com/steveyen/gkvlite/blob/master/collection.go

It just randomly walks down one tree branch and evicts what it finds. It also returns the count of items evicted so you have a sense of how many times you should call it.

Or, if you really want it just mostly on disk, just close the gkvlite (after Flush()) and reopen it up to let GC reclaim things.