memory leak?

[dim]

I am not sure if it's the C code or the go wrapper or maybe something I misunderstand altogether, but the following piece of code is an easy way to consume all available memory of your machine:

package main

import (
    "encoding/binary"
    "log"

    fdb "github.com/couchbaselabs/goforestdb"
)

func abortOn(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

func main() {
    db, err := fdb.Open("/tmp/leak.fdb", fdb.DefaultConfig())
    abortOn(err)

    for i := 0; i < 100000000; i++ {
        key := make([]byte, 8)
        binary.BigEndian.PutUint64(key, uint64(i))
        err := db.SetKV(key, []byte("BOGUSDATA"))
        abortOn(err)

        if (i+1)%1000000 == 0 {
            log.Printf("written %d keys", i+1)
        }
    }

}

It looks like a memory leak to me. I tried both, the db.Set and the db.SetKV methods (obviously closing the documents) and they both produce the same pattern of uncontrolled memory consumption growth.

Any ideas? Thanks!

[mschoch]

I'm not able to reproduce this issue. I just ran it on my Macbook Air and it ran to completion in about 10 minutes. It generally had between 500Mb and 1GB resident memory the whole time.

$ ./test-fdb 
2014/10/30 08:09:29 written 1000000 keys
2014/10/30 08:09:32 written 2000000 keys
2014/10/30 08:09:37 written 3000000 keys
2014/10/30 08:09:41 written 4000000 keys
2014/10/30 08:09:46 written 5000000 keys
2014/10/30 08:09:51 written 6000000 keys
2014/10/30 08:09:56 written 7000000 keys
2014/10/30 08:10:00 written 8000000 keys
2014/10/30 08:10:05 written 9000000 keys
2014/10/30 08:10:10 written 10000000 keys
2014/10/30 08:10:15 written 11000000 keys
2014/10/30 08:10:21 written 12000000 keys
2014/10/30 08:10:26 written 13000000 keys
2014/10/30 08:10:31 written 14000000 keys
2014/10/30 08:10:36 written 15000000 keys
2014/10/30 08:10:41 written 16000000 keys
2014/10/30 08:10:46 written 17000000 keys
2014/10/30 08:10:51 written 18000000 keys
2014/10/30 08:10:56 written 19000000 keys
2014/10/30 08:11:01 written 20000000 keys
2014/10/30 08:11:07 written 21000000 keys
2014/10/30 08:11:12 written 22000000 keys
2014/10/30 08:11:17 written 23000000 keys
2014/10/30 08:11:23 written 24000000 keys
2014/10/30 08:11:28 written 25000000 keys
2014/10/30 08:11:33 written 26000000 keys
2014/10/30 08:11:39 written 27000000 keys
2014/10/30 08:11:45 written 28000000 keys
2014/10/30 08:11:50 written 29000000 keys
2014/10/30 08:11:55 written 30000000 keys
2014/10/30 08:12:00 written 31000000 keys
2014/10/30 08:12:06 written 32000000 keys
2014/10/30 08:12:11 written 33000000 keys
2014/10/30 08:12:16 written 34000000 keys
2014/10/30 08:12:22 written 35000000 keys
2014/10/30 08:12:27 written 36000000 keys
2014/10/30 08:12:33 written 37000000 keys
2014/10/30 08:12:40 written 38000000 keys
2014/10/30 08:12:45 written 39000000 keys
2014/10/30 08:12:51 written 40000000 keys
2014/10/30 08:12:56 written 41000000 keys
2014/10/30 08:13:02 written 42000000 keys
2014/10/30 08:13:11 written 43000000 keys
2014/10/30 08:13:19 written 44000000 keys
2014/10/30 08:13:27 written 45000000 keys
2014/10/30 08:13:34 written 46000000 keys
2014/10/30 08:13:40 written 47000000 keys
2014/10/30 08:13:47 written 48000000 keys
2014/10/30 08:13:53 written 49000000 keys
2014/10/30 08:14:00 written 50000000 keys
2014/10/30 08:14:05 written 51000000 keys
2014/10/30 08:14:11 written 52000000 keys
2014/10/30 08:14:17 written 53000000 keys
2014/10/30 08:14:23 written 54000000 keys
2014/10/30 08:14:29 written 55000000 keys
2014/10/30 08:14:35 written 56000000 keys
2014/10/30 08:14:41 written 57000000 keys
2014/10/30 08:14:47 written 58000000 keys
2014/10/30 08:14:53 written 59000000 keys
2014/10/30 08:14:59 written 60000000 keys
2014/10/30 08:15:05 written 61000000 keys
2014/10/30 08:15:10 written 62000000 keys
2014/10/30 08:15:17 written 63000000 keys
2014/10/30 08:15:24 written 64000000 keys
2014/10/30 08:15:31 written 65000000 keys
2014/10/30 08:15:37 written 66000000 keys
2014/10/30 08:15:44 written 67000000 keys
2014/10/30 08:15:50 written 68000000 keys
2014/10/30 08:15:56 written 69000000 keys
2014/10/30 08:16:02 written 70000000 keys
2014/10/30 08:16:08 written 71000000 keys
2014/10/30 08:16:14 written 72000000 keys
2014/10/30 08:16:21 written 73000000 keys
2014/10/30 08:16:27 written 74000000 keys
2014/10/30 08:16:32 written 75000000 keys
2014/10/30 08:16:38 written 76000000 keys
2014/10/30 08:16:44 written 77000000 keys
2014/10/30 08:16:50 written 78000000 keys
2014/10/30 08:16:56 written 79000000 keys
2014/10/30 08:17:02 written 80000000 keys
2014/10/30 08:17:08 written 81000000 keys
2014/10/30 08:17:14 written 82000000 keys
2014/10/30 08:17:20 written 83000000 keys
2014/10/30 08:17:26 written 84000000 keys
2014/10/30 08:17:32 written 85000000 keys
2014/10/30 08:17:38 written 86000000 keys
2014/10/30 08:17:44 written 87000000 keys
2014/10/30 08:17:49 written 88000000 keys
2014/10/30 08:17:56 written 89000000 keys
2014/10/30 08:18:03 written 90000000 keys
2014/10/30 08:18:09 written 91000000 keys
2014/10/30 08:18:16 written 92000000 keys
2014/10/30 08:18:22 written 93000000 keys
2014/10/30 08:18:28 written 94000000 keys
2014/10/30 08:18:34 written 95000000 keys
2014/10/30 08:18:41 written 96000000 keys
2014/10/30 08:18:47 written 97000000 keys
2014/10/30 08:18:52 written 98000000 keys
2014/10/30 08:18:58 written 99000000 keys
2014/10/30 08:19:04 written 100000000 keys

Can you provide additional details about the system where you see the problem? It would be helpful to know the hardware/OS/go version etc. I think with the default configuration ForestDB maintains its own cache, its possible this just needs to be tuned better for your environment.

[dim]

My workstation is Ubuntu 14.04 x86_64 with Go 1.3.3. My memory consumption jumps to 2G within seconds and the process is killed by the Kernel within a minute.

I have tried to run it on Debian Wheezy too, on a machine with 24G of memory. It took slightly longer, but the result was the same.

SHAs:

• https://github.com/couchbaselabs/goforestdb/commit/b4c2a74ffc29f07775adf4c2a99ecf1953300a89
• https://github.com/couchbaselabs/forestdb/tree/f3dd5a089ca1d9136077f35c479fd416895dab20

[mschoch]

OK, I'll give it a try on Ubuntu this afternoon.

[mschoch]

Confirmed it behaves very differently on Ubuntu:

...
2014/10/30 13:52:06 written 57000000 keys
2014/10/30 13:52:10 written 58000000 keys
2014/10/30 13:52:13 written 59000000 keys
2014/10/30 13:52:18 written 60000000 keys
Killed

I guess the next step is for me to write the equivalent C version and see how it behaves. I'm hoping its something in ForestDB itself, because the calls we make are pretty straightforward.

[mschoch]

OK, I'm not much of a C guy, but here is what I think is an equivalent C implementation:

#include <stdio.h>
#include <string.h>
#include <assert.h>
#include "libforestdb/forestdb.h"

int main(int argc, char**argv) {

    fdb_handle *db;
    fdb_status status;
    fdb_config config;

    config = fdb_get_default_config();
    status = fdb_open(&db, "/tmp/db", &config);
    assert(status == FDB_RESULT_SUCCESS);
    printf("opened database\n");

    int i;
    for (i = 0; i < 100000000; i++) {
        char key[10];
        snprintf(key, 9, "%d", i);
        int keylen = strlen(key);

        status = fdb_set_kv(db, key, keylen, "BOGUSDATA", 9);
        assert(status == FDB_RESULT_SUCCESS);

        if((i+1)%1000000 == 0) {
            printf("written %d keys\n", i+1);
        }
    }
}

This also consumes all memory and is killed on Ubuntu. Next I'll try it on the Mac just to see how it works there, then I'll pass this up the chain to the forestdb team.

[mschoch]

It just occurred to me that it might be because you're never calling fdb_commit. Its still unusual that it works without crash on mac and not linux, but when I changed the last block to:

        if((i+1)%1000000 == 0) {
            status = fdb_commit(db, FDB_COMMIT_NORMAL);
            assert(status == FDB_RESULT_SUCCESS);
            printf("written %d keys\n", i+1);
        }

Its not done yet, but the memory usage of the process is much lower and remaining constant.

The equivalent change to the Go program would be to change the last block to:

        if (i+1)%1000000 == 0 {
            err = db.Commit(COMMIT_NORMAL)
            abortOn(err)
            log.Printf("written %d keys", i+1)
        }

[mschoch]

OK, so you have 2 options:

1. invoke Commit() periodically
2. on the configuration, set cfg.SetWalFlushBeforeCommit(true)

I have tested both on Ubuntu and they stay within a reasonable amount of RAM.

I also started a discussion on the ForestDB group around the behavior when Commit() is not called periodically. Also, if you think the default configuration should be different please join the conversation here:

https://groups.google.com/d/msg/forestdb/qh9DuNez6kM/52YyF7l-bcsJ