Memory usage jumps on start

[martinsookael]

I'm using a ~95 MB database

var Datastore = require("nedb")    
var foo = new Datastore({ filename: "./db/foo.jsonl", autoload: true })

When I start the app, the memory usage jumps to 1.1 GB and stays there until I make the first request to db and then falls down to under 300 MB.

Has anyone experienced something like this?

[gustavo-oliveira-mendonca]

@martinsookael I'm making some load tests in my application and found the same problem. My RAM usage jumps to over 4.0GB! Working with a 20k DB of large documents (.db with 200MB). Although this is not the real scenario that the application will work, it's certainly something that worries me. I understand that the entire DB is loaded in memory, but I belive that's not the complete reason for the huge amount of memory.

Memory footprint

A copy of the whole database is kept in memory. This is not much on the expected kind of datasets (20MB for 10,000 2KB documents).

I was able to take a heap snapshot after a few attemps and it shows millions of objects containing my documents, each with 27kb. I'm trying to work in it to see if the objects are loaded multiple times. I'll inform here anything I found.

[gustavo-oliveira-mendonca]

472

[marcusjwhelan]

@Noctua36 @martinsookael #506 . Every index duplicates every document. In response to this issue myself and issuer of 506 created TeDB. Depending on your use case you can write a special Storage Driver for it.