JavaScript heap out of memory during validation

[chrisfleisch]

I've been using snap-to-s3 for the last couple of years with any issues like this. I'm now seeing a JavaScript memory issue on some snapshots during the validation step. I tried increasing the instance size from a c5.large to r5.large, but that didn't seem to help. Do you know what I could do to resolve this issue? There's the full error message:

(node:3259) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use node --trace-warnings ... to show where the warning was created)
[snap-0bc4fff30da0bb123] Migrating snap-0bc4fff30da0bb123 to S3
[snap-0bc4fff30da0bb123] Tagging snapshot with "migrating"...
[snap-0bc4fff30da0bb123] Creating temporary EBS volume of type "gp2" from snapshot
[snap-0bc4fff30da0bb123] Attaching vol-00cbce6055d9e7123 to this instance (i-0e9a57a184b904123) at /dev/sdj...
[snap-0bc4fff30da0bb123] Waiting for vol-00cbce6055d9e7123's partitions to become visible to the operating system...
[snap-0bc4fff30da0bb123] 1 partition to upload
[snap-0bc4fff30da0bb123]
[snap-0bc4fff30da0bb123] Uploading partition 1 of 1...
[snap-0bc4fff30da0bb123] Mounting /dev/nvme1n1p1 at /mnt/snap-0bc4fff30da0bb123-p1...
[snap-0bc4fff30da0bb123] Computing size of files to upload...
[snap-0bc4fff30da0bb123] 764.57 GB to compress and upload to s3://my-infra-snap-to-s3-archive/vol-0d7f97a969a191123/2023-07-12T06:24:06+00:00 snap-0bc4fff30da0bb123 - Created for policy: policy-0fb98236512681123 schedule: daily_1096 at 0600.p1.tar.zstd
[snap-0bc4fff30da0bb123] Progress is based on the pre-compression data size:
[snap-0bc4fff30da0bb123] Upload complete, now validating the upload of this partition...
<--- Last few GCs --->
[3259:0x7126da0] 169032912 ms: Mark-sweep 1896.1 (2094.1) -> 1889.7 (2086.9) MB, 1856.9 / 0.0 ms  (average mu = 0.339, current mu = 0.092) allocation failure; scavenge might not succeed
[3259:0x7126da0] 169035403 ms: Mark-sweep 1905.5 (2086.9) -> 1897.6 (2107.6) MB, 2335.3 / 0.0 ms  (average mu = 0.205, current mu = 0.063) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7b3e0 node::Abort() [node]
 2: 0xa8c8aa  [node]
 3: 0xd69100 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd694a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf46ba5  [node]
 6: 0xf5908d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf3378e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf34b57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf15d2a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12dacdf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1707b79  [node]

[thenickdude]

Do you know how many files are on the volume? Millions, or billions?

[chrisfleisch]

It's a little less than 5 million files. Total size is about 813G.

[chrisfleisch]

I was able to get around this error by providing node more than the 2GB of default heap memory. I changed my command to include an env variable for node:

sudo NODE_OPTIONS="--max-old-space-size=16000" snap-to-s3

I then edited index.js to log the heap stats to confirm the setting change:

const v8 = require('node:v8');
console.log(v8.getHeapStatistics());