Random Heap overflows

[antoniojtorres]

Hello, first of all thanks for the great package, it works wonderfully.

I think I'm having a strange issue and suspect the root may be this package. I am using puppeteer to crawl through sites so I'm doing the typical breath-first crawl where I essentially crawl in a loop, go from one page to the next by clicking in the same session. After a few hours of this happening, without fail, the crawler crashes with a heap OOM issue. I'm adding it below.

I use

const cursor = createCursor(page);  
await cursor.click(linkToClick);

What's strange is that there are no other errors and node's own heap seems totally stable up until this happens, unless I'm missing how to accurately calculate this. I'm using process.memoryUsage(); to get the info out.

I am working on a minimal repro now, but it takes hours to make it happen so I'll be back. Just posting this in case anyone has had this experience before.

Thanks in advance for any advice.

[18:0x49a3f30]   142189 ms: Scavenge 1031.6 (1069.2) -> 1031.6 (1069.2) MB, 93.2 / 0.0 ms  (average mu = 0.997, current mu = 0.800) allocation failure
[18:0x49a3f30]   144223 ms: Mark-sweep 1605.4 (1643.0) -> 991.0 (1029.1) MB, 610.5 / 0.0 ms  (+ 0.8 ms in 1 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 2034 ms) (average mu = 0.978, current mu = 0.828) allocation fai
<--- JS stacktrace --->

==== JS stack trace =========================================
    0: ExitFrame [pc: 0x140de99]

Security context: 0x31dfc86808d1 <JSObject>

    1: getLUT [0x25054c602cc9] [/node_modules/bezier-js/lib/bezier.js:~249] [pc=0x9b6638e9d17](this=0x1f5332321ba1 <Bezier map = 0x337528555069>,0x05e0a6fbd7d1 <HeapNumber inf>)
    2: path(aka path) [0x1a1490b50791] [/node_modules/ghost-cursor/lib/spoof.js:187] [bytecode=0x153fbe7bb171 offset=164](this=0x2429bc0804b1 <undefined>,0x1f5332321cb1 <Object map = 0x1f...

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
 1: 0xa1a640 node::Abort() [node]
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9a68e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9aa09 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd57c85  [node]
 6: 0xd2fc70 v8::internal::Factory::NewUninitializedFixedArray(int, v8::internal::AllocationType) [node]
 7: 0xe9f161  [node]
 8: 0xe9f410  [node]
 9: 0x1047ddb v8::internal::Runtime_GrowArrayElements(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x140de99  [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! crawler@1.0.0 start: `node crawler.js`
npm ERR! Exit status 134
npm ERR!
npm ERR! Failed at the ua-data-crawler@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/pptruser/.npm/_logs/2021-04-26T19_39_00_063Z-debug.log

[Niek]

Strange, which Node version are you using? Did you try with a newer Node version already?

In any case, the bezierjs dependency definitely needs to be bumped, but I couldn't figure out how to get it to play nicely with TS. See also: https://github.com/DefinitelyTyped/DefinitelyTyped/issues/49185

[jaean123]

I found that this issue appears when using the mouse helper. Try disabling that.

[Niek]

@antoniojtorres can you confirm you were using the mouse helper too? It might be related to the random movements too, did you use performRandomMoves?

[jaean123]

I apologize for the mistake. I have confirmed that this bug appears when using the mouse helper. It's coincidence that it did not appear on multiple attempts after disabling mouse helper.

[Niek]

@jaean123 do you have a simple test script to reproduce this? I have a hard time replicating the issue.

[jaean123]

@Niek I'll see if I can come up with a script. Thanks for your continued support for this project.

[precious-void]

@Niek reproduced it on visible input with { width: 0, height: 0 }, might happen because of devision to zero, or another weird staff.

[Niek]

@Niek reproduced it on visible input with { width: 0, height: 0 }, might happen because of devision to zero, or another weird staff.

@shtelzerartem can you please upload a minimal sample here so I can check?

[Niek]

See #104

This should be solved in v1.1.19 - feel free to re-open of it still happens.