search

attic-labs / noms

The versioned, forkable, syncable database
Apache License 2.0
7.45k stars 266 forks source link

Sync: Avoid exploding memory with chunk data #3692

Closed cmasone-attic closed 6 years ago

cmasone-attic commented 6 years ago

After #3659, we need to avoid exploding memory by attempting to load too many chunks of a given level at once.

ghost commented 6 years ago

I think this is now a GoodFirstBug. Just put a maximum batchSize for processing each level and make an inner loop which works in maximum batchSize quanta.