Javascript heap out of memory error

[mkieselmann]

Hi, I'm self-hosting powersync on AWS and connect to a large RDS database with many changes happening at a time. Today I repeatedly ran into the following error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Stack trace:

0x7f4ec6099ef6
11: 0x1537e36 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x10ffa55 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10fe8c1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x11285bc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x1127da4 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
5: 0x1110eb4 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
4: 0x1110925  [node]
3: 0xefef17 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
2: 0xefec30 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
----- Native stack trace -----
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
<--- JS stacktrace --->
[1:0x6cbf6a0]   214567 ms: Scavenge (reduce) 2046.4 (2082.1) -> 2045.8 (2082.3) MB, 51.50 / 0.00 ms  (average mu = 0.350, current mu = 0.321) allocation failure;
[1:0x6cbf6a0]   214435 ms: Scavenge (reduce) 2046.3 (2082.1) -> 2045.6 (2082.1) MB, 52.61 / 0.00 ms  (average mu = 0.350, current mu = 0.321) allocation failure;
[1:0x6cbf6a0]   214316 ms: Scavenge (reduce) 2046.1 (2081.8) -> 2045.5 (2082.1) MB, 57.46 / 0.00 ms  (average mu = 0.350, current mu = 0.321) allocation failure;
<--- Last few GCs --->

Seems like the service started replication and ran out of memory.

On the database side I see the following query beeing executed: SELECT * FROM pg_catalog.pg_logical_slot_peek_binary_changes($1, ?, ?, ?, ?, ?, $2)

[mkieselmann]

Increasing the available heap size for node seems to fix the issue: ENTRYPOINT ["node", "--max-old-space-size=8192", "service/lib/entry.js"]

Addition @rkistner pointed out how to set max-old-space-size:

Configure an env var NODE_OPTIONS=--max-old-space-size=800. This example is for 800mb, but set it to around 80% of available memory