JavaScript heap out of memory

[ptorrent]

Hello,

Trying this tool on my server:

30G RAM free 24CPU HolterTower.gltf is 460M

node --max-old-space-size=16192 ./node_modules/.bin/xeokit-convert -s HolterTower.gltf -m HolterTower.json -o test2.xkt

<--- Last few GCs --->

[12770:0x6043e80]     7391 ms: Scavenge 745.2 (778.8) -> 745.2 (778.8) MB, 137.0 / 0.0 ms  (average mu = 0.930, current mu = 0.920) allocation failure

<--- JS stacktrace --->

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory

Is there a way to solve this ?

[ptorrent]

Same issue with an other gltf file 288M

[xeolabs]

Not sure what can be done to xeokit-convert, unless there is a memory leak we need to fix (not aware of any at this time).

Otherwise, it seems to be more related to Node memory management, or possibly a Node configuration?

[ptorrent]

Don't think so I did some test with this script:

const array = [];
while (true) {
        // This makes the array bigger on each iteration
         array.push(new Array(10000000));

         const memory = process.memoryUsage();
         console.log((memory.heapUsed / 1024 / 1024 / 1024).toFixed(4), 'GB')
}

node --max-old-space-size=16000 myscript.js

The maximum memory usage for a script is

15.4271 GB
15.5016 GB
15.5762 GB

and for --max-old-space-size=8192:

7.8274 GB
7.9019 GB
7.9764 GB

Looks like working but maybe not with

node --max-old-space-size=8192 ./node_modules/.bin/xeokit-convert

[ptorrent]

./node-v16.17.0-linux-x64/bin/node --max-old-space-size=8192 ./node_modules/.bin/xeokit-convert.js -s svc.gltf -m svc.json -o svc.xkt

<--- Last few GCs --->

[28963:0x68b9c40]     4619 ms: Scavenge 426.5 (460.2) -> 426.5 (460.2) MB, 62.4 / 0.0 ms  (average mu = 0.989, current mu = 0.989) allocation failure
[28963:0x68b9c40]     8292 ms: Mark-sweep 810.5 (844.2) -> 607.2 (641.7) MB, 414.3 / 0.0 ms  (+ 0.1 ms in 2 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 5297 ms) (average mu = 0.946, current mu = 0.940) allocation fai

<--- JS stacktrace --->

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: 0xb02930 node::Abort() [./node-v16.17.0-linux-x64/bin/node]
 2: 0xa18149 node::FatalError(char const*, char const*) [./node-v16.17.0-linux-x64/bin/node]
 3: 0xcdd16e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [./node-v16.17.0-linux-x64/bin/node]
 4: 0xcdd4e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [./node-v16.17.0-linux-x64/bin/node]
 5: 0xe94b55  [./node-v16.17.0-linux-x64/bin/node]
 6: 0x10c0d5d  [./node-v16.17.0-linux-x64/bin/node]
 7: 0x10c0f33 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::HashTable<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, int, v8::internal::AllocationType) [./node-v16.17.0-linux-x64/bin/node]
 8: 0x10c1574 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::Dictionary<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [./node-v16.17.0-linux-x64/bin/node]
 9: 0xfe94c8  [./node-v16.17.0-linux-x64/bin/node]
10: 0x106f795 v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [./node-v16.17.0-linux-x64/bin/node]
11: 0x10b34ee v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [./node-v16.17.0-linux-x64/bin/node]
12: 0x106a681 v8::internal::JSObject::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>) [./node-v16.17.0-linux-x64/bin/node]
13: 0x106d54f v8::internal::JSReceiver::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>) [./node-v16.17.0-linux-x64/bin/node]
14: 0x11ef301 v8::internal::Runtime_CreateDataProperty(int, unsigned long*, v8::internal::Isolate*) [./node-v16.17.0-linux-x64/bin/node]
15: 0x15d5439  [./node-v16.17.0-linux-x64/bin/node]
Aborted

I'm not expert in memory heap, but looks more an issue about a loop calling a function. My memory state on the server is correct max ~6%. But my CPU 120%

[ptorrent]

With some logs (-l)

[convert2xkt] Running convert2xkt v1.1.0...
[convert2xkt] Reading input file: svc.gltf
[convert2xkt] Input file size: 288709.84 kB
[convert2xkt] Reading input metadata file: svc.json

<--- Last few GCs --->

[30179:0x519bca0]     3772 ms: Scavenge 426.4 (460.5) -> 426.4 (460.5) MB, 65.6 / 0.0 ms  (average mu = 0.976, current mu = 0.976) allocation failure
[30179:0x519bca0]     7384 ms: Mark-sweep 810.4 (844.5) -> 607.2 (641.5) MB, 434.8 / 0.0 ms  (+ 87.9 ms in 2 steps since start of marking, biggest step 87.9 ms, walltime since start of marking 5146 ms) (average mu = 0.924, current mu = 0.921) allocation f

<--- JS stacktrace --->

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: 0xb02930 node::Abort() [./node-v16.17.0-linux-x64/bin/node]
 2: 0xa18149 node::FatalError(char const*, char const*) [./node-v16.17.0-linux-x64/bin/node]
 3: 0xcdd16e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [./node-v16.17.0-linux-x64/bin/node]
 4: 0xcdd4e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [./node-v16.17.0-linux-x64/bin/node]
 5: 0xe94b55  [./node-v16.17.0-linux-x64/bin/node]
 6: 0x10c0d5d  [./node-v16.17.0-linux-x64/bin/node]
 7: 0x10c0f33 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::HashTable<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, int, v8::internal::AllocationType) [./node-v16.17.0-linux-x64/bin/node]
 8: 0x10c1574 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::Dictionary<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [./node-v16.17.0-linux-x64/bin/node]
 9: 0xfe94c8  [./node-v16.17.0-linux-x64/bin/node]
10: 0x106f795 v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [./node-v16.17.0-linux-x64/bin/node]
11: 0x10b34ee v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [./node-v16.17.0-linux-x64/bin/node]
12: 0x106a681 v8::internal::JSObject::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>) [./node-v16.17.0-linux-x64/bin/node]
13: 0x106d54f v8::internal::JSReceiver::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>) [./node-v16.17.0-linux-x64/bin/node]
14: 0x11ef301 v8::internal::Runtime_CreateDataProperty(int, unsigned long*, v8::internal::Isolate*) [./node-v16.17.0-linux-x64/bin/node]
15: 0x15d5439  [./node-v16.17.0-linux-x64/bin/node]
Aborted

[ptorrent]

Looks better when I use:

./node-v16.17.0-linux-x64/bin/node --max-old-space-size=25192 ./node_modules/@xeokit/xeokit-convert/convert2xkt.js -s test.ifc -o test.ifc.xkt -l

[xeolabs]

@ptorrent is it working OK now?

I'll add that as a hint in the docs, thanks

[ptorrent]

A bit better (it seems to process works hard) but.. :

https://github.com/xeokit/xeokit-convert/issues/72

[xeolabs]

I just published an optimization in xeokit-convert 1.1.3 that might help with this issue - see #78

[xeolabs]

A resource about nodejs memory limits: https://blog.appsignal.com/2021/12/08/nodejs-memory-limits-what-you-should-know.html