levelDB out of memory

[GoogleCodeExporter]

What steps will reproduce the problem?
I make a 32-bit app using levelDB in VS2010 sp1 in windows 7 64bit.I run this 
app in Windows 2008 Server R2, after about 1 hour, this app raise 
stl::bad_alloc exception in string::append function.I find Virtual bytes of 
this app is about 2G.

key is 12 bytes
value is about 200-300 bytes

kxTargetFileSize = 32M

write radio about 4M/s(random write)

levelnum = 3
no compression
writebuffersize=256M

this is bug or memory fragments

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?
levelDB 1.4 in windows 2008 Server r2 64bit

Please provide any additional information below.

Original issue reported on code.google.com by david...@163.com on 10 May 2012 at 8:44

[GoogleCodeExporter]

> kxTargetFileSize = 32M
> levelnum = 3

The leveldb package provides no way for callers to change these parameters
without editing the source code. We can only help with problems if your
problem arises with an unmodified leveldb.

> writebuffersize=256M

Why such a large write buffer size? Note that during compactions two write
buffers are live, so this will immediately use up 512MB, which is a pretty
big chunk of the memory available to a 32-bit process.

> this is bug or memory fragments

It is unlikely to be a leak (we run all of our tests with gperftool's leak
detection mode turned on). It could be fragmentation as you suggest, or it
could be caused by other sources of memory usage (e.g., each open Table
file's index is loaded into memory).

If you have a short standalone program that reproduces this (with an
unmodified leveldb), we could try running it in a mode where we can collect
more memory usage statistics.

Original comment by san...@google.com on 10 May 2012 at 3:09

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

I meet the same question with the descripton of Issue 90.

I use the leveldb default project which get out from:
http://code.google.com/p/leveldbwin/downloads/detail?name=leveldb_1_20_win32_src
.zip&can=2&q= 

1. not define DB_BENCH;
2. not define USE_VISTA_API;
3. using the following code:

leveldb::DB* g_db = NULL;
void WriteDb(leveldb::DB* db, int Index)
{
    const char* Content = "We now decided that after three years it might"
        " be a good idea to re-assess our current situation by another,"
        " more extensive survey. It is similar to the original one "
        "(since we included the old questions to be able to compare the results),"
        " but the survey also contains a bunch of new ones to take new developments"
        " such as Ogre usage on mobile devices into account.";
    char Buff[32];
    ZeroMemory(Buff,sizeof(Buff));
    leveldb::WriteOptions wo;
    wo.sync = false;
    for(int i = 0 ; i < 20000 ;i++)
{
sprintf_s(Buff, 32, "Name%d-%d",Index, i);

        leveldb::Status s = db->Put(wo,Buff,Content);
    }
}

void tt(int Index)
{
     assert( status.ok() );
    {
        uint64_t tickbegin = GetTickCount64();
        WriteDb(g_db,Index);
        uint64_t tickend = GetTickCount64();
        cout << "write 20000 in " << tickend - tickbegin  << " Milliseconds"<< endl;
    }
}

int _tmain(int argc, _TCHAR* argv[])
{
  leveldb::Options options;
options.create_if_missing = true;
options.write_buffer_size = 8 * 1024* 1024;
leveldb::Status status = leveldb::DB::Open(options, "c:/tmp/testdb", &g_db);

int repeat = 30000;
    for(int i = 0 ; i < repeat ; i++){
        tt(i);
    } 
    cout << repeat << " times done."<< endl;
   delete g_db;;
    return _getch();
}

build and run "test" app, then will be found the process's virtual bytes 
increase all the time.

Others:
operating system: Windows XP SP3.
32-bit app in VS2010.
LevelDB Version:1.20 , get out from "http://code.google.com/p/leveldbwin/".

Original comment by duanever...@163.com on 17 May 2012 at 1:18

• Added labels: ****
• Removed labels: ****