Web app: out of memory when submitting remaining observations

[pheyvaer]

When submitting remaining observation via the the Web app I get the following error in the engine:

2023-04-26T09:22:35.908Z [WEB API] info: We're going to replay the REMAINING observations and their related information from the current pointer onwards: 4
2023-04-26T09:22:35.909Z [WEB API] info: Retrieving metadata of LDESinLDP if it already exists.
2023-04-26T09:22:35.958Z [WEB API] info: Retrieving all related information to the Observation being replayed.

<--- Last few GCs --->

[96973:0x7fd8ad94d000]  1064676 ms: Mark-sweep 4051.5 (4137.1) -> 4038.2 (4139.9) MB, 1048.3 / 0.0 ms  (average mu = 0.120, current mu = 0.018) allocation failure; scavenge might not succeed
[96973:0x7fd8ad94d000]  1065761 ms: Mark-sweep 4054.1 (4139.9) -> 4040.7 (4142.4) MB, 914.5 / 0.0 ms  (average mu = 0.138, current mu = 0.157) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10de27d55 node::Abort() (.cold.1) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 2: 0x10c8f0259 node::Abort() [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 3: 0x10c8f043e node::OOMErrorHandler(char const*, bool) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 4: 0x10ca6b193 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 5: 0x10cc33bd5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 6: 0x10cc380a0 v8::internal::Heap::CollectSharedGarbage(v8::internal::GarbageCollectionReason) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 7: 0x10cc348ef v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 8: 0x10cc319c8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
 9: 0x10cc248ea v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
10: 0x10cc25265 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
11: 0x10cc076de v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
12: 0x10d03d7ec v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
13: 0x10d437df9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/pheyvaer/.nvm/versions/node/v18.12.1/bin/node]
sh: line 1: 96973 Abort trap: 6           node dist/app.js

Do you expect the engine to need so much memory?

[svrstich]

This is a known issue ... happens to larger datasets. We have been exploring manners to overcome this issue, but still need to implement them. https://gitlab.ilabt.imec.be/svrstich/ldes-in-solid-semantic-observations-replay/-/issues/1

[pheyvaer]

Ok, this is not a deal breaker for me then. Do you have a link to a dataset that should not have this issue in most cases?

[svrstich]

https://cloud.ilabt.imec.be/index.php/s/8BatNcg2iEyJktR

[pheyvaer]

That works! Is it ok to mention this dataset in the README and use it for the whole flow?

[svrstich]

Of course, this is only a slimmed-down selection of the entire participant 1 dataset.

[argahsuknesib]

Hello, I am wondering if this issue was resolved because I still face the issue when loading up heavier datasets with the engine. Can you help me with this? @svrstich

[svrstich]

Hi Kush, I'm in iGent tomorrow. Have you tried the latest version, as the replay with the remaining observations should now be using a batch kind approach, so that the OoM should not happen anymore.

[argahsuknesib]

Sure, I tried the latest version. But I will try it again. Otherwise, will see you tomorrow :)