Report crashes - Allocation failed - JavaScript heap out of memory

[mgifford]

If I crawl this sitemap.xml file https://cnib.ca/en/sitemap.xml?region=on

The page fails. There are over 1800 pages in it, but had hoped it would be able to manage sites larger than this.

INFO: BasicCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
INFO: Crawler final request statistics: {"avgDurationMillis":8803,"perMinute":34,"finished":1813,"failed":3,"retryHistogram":[1811,2,null,3]}
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[9364:0x102679000]  3255724 ms: Mark-sweep 1306.0 (1388.6) -> 1305.9 (1357.1) MB, 68.6 / 0.0 ms  (average mu = 0.719, current mu = 0.000) last resort GC in old space requested
[9364:0x102679000]  3255797 ms: Mark-sweep 1305.9 (1357.1) -> 1305.9 (1357.1) MB, 72.7 / 0.0 ms  (average mu = 0.556, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x200b585878a1]
Security context: 0x1c9e2dd1e6e1 <JSObject>
    1: byteLength(aka byteLength) [0x1c9eb2305ef1] [buffer.js:~510] [pc=0x200b58ca5d2c](this=0x1c9e693826f1 <undefined>,string=0x1c9e2920a331 <Very long string[460960089]>,encoding=0x1c9e2dd3d4f1 <String[4]: utf8>)
    2: arguments adaptor frame: 3->2
    3: fromString(aka fromString) [0x1c9eb231c639] [buffer.js:~335] [pc=0x200b58cec790](this=0x1c9e693826f1 ...

 1: 0x10003ae75 node::Abort() [/usr/local/bin/node]
 2: 0x10003b07f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001a7ae5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100572ef2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x10057c3f4 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 6: 0x10054e1e4 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 7: 0x10067fd99 v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node]
 8: 0x1001c587d v8::String::Utf8Length() const [/usr/local/bin/node]
 9: 0x10004e7b6 node::Buffer::(anonymous namespace)::ByteLengthUtf8(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
10: 0x200b585878a1 
The file /Users/mikegifford/purple-hats/results/2020-9-24/160096044136e10467e2/reports/report.html does not exist.

What's the best way to deal with this memory error

[younglim]

No longer replicated on current PlaywrightCrawler / Playwright used in Purple A11y

[younglim]

Hi @mgifford,

I encountered the Out of Memory issue when making large scans > thousands of pages.

I have increased the limit of node when running npm run cli and npm run start to allocate up to 6GB (it will only consume then error out if it reaches that much memory usage).

Refer to https://github.com/GovTechSG/purple-a11y/commit/d4c3b94e293b667aa62b2cf70148ace3c812eb00.

Hope this helps!