techfort / LokiJS

javascript embeddable / in-memory database
http:/techfort.github.io/LokiJS
MIT License
6.74k stars 482 forks source link

Maximum number of objects #890

Closed bcatalin closed 3 years ago

bcatalin commented 3 years ago

The problem was from V8 heap. Increasing the heap from 4 to 16 was the solution.

What is the maximum number of objects that can be inserted in a collection ?

I've tested it on a 32GB system and on 10 millions the V8 crashed. At that time the RAM was around 6GB occupied.

<--- Last few GCs --->

[7690:0x5394a90] 112698 ms: Scavenge (reduce) 4094.3 (4100.5) -> 4093.9 (4101.8) MB, 5.3 / 0.0 ms (average mu = 0.108, current mu = 0.045) allocation failure [7690:0x5394a90] 115796 ms: Mark-sweep (reduce) 4094.6 (4103.8) -> 4093.1 (4104.0) MB, 3094.9 / 0.0 ms (+ 0.1 ms in 42 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3121 ms) (average mu = 0.066, current mu = 0.021)

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xa25510 node::Abort() [node] 2: 0x9664d3 node::FatalError(char const, char const) [node] 3: 0xb9a8ee v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xb9ac67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xd56cd5 [node] 6: 0xd5785f [node] 7: 0xd6569b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 8: 0xd6925c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node] 9: 0xd3793b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node] 10: 0x107fc1f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long, v8::internal::Isolate) [node] 11: 0x1426939 [node]