FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - Javascript heap out of memory

[alesanu]

Hello

I'm migrating Nightwatch from v0.9 to 1.0.18 (1.0.19 today) and I got an error after migration when I run a lot of tests (300 cases) on Jenkins. I've tried to increase max_old_space_size, to clear the garbage collector but no results.

I'll paste the output here:

[8780:0x35b15b0]  5994849 ms: Mark-sweep 1358.7 (1463.3) -> 1358.6 (1463.3) MB, 1468.2 / 0.0 ms  allocation failure GC in old space requested
[8780:0x35b15b0]  5996269 ms: Mark-sweep 1358.6 (1463.3) -> 1358.6 (1432.3) MB, 1413.3 / 0.0 ms  last resort GC in old space requested
[8780:0x35b15b0]  5997723 ms: Mark-sweep 1358.6 (1432.3) -> 1358.6 (1432.3) MB, 1453.4 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35163f3258b9 <JSObject>
    1: onread [net.js:~583] [pc=0x1649d5c47507](this=0xa88c6a4db71 <TCP map = 0x3f2c8c220449>,nread=398,buffer=0xa88c6a4f369 <Uint8Array map = 0x1fe00edf5429>)
    2: arguments adaptor frame: 3->2

==== Details ================================================

[1]: onread [net.js:~583] [pc=0x1649d5c47507](this=0xa88c6a4db71 <TCP map = 0x3f2c8c220449>,nread=398,buffer=0xa88c6a4f369 <Uint8Array ma...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x8ccf9c [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [node]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
 7: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
 8: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
 9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x1649d54042fd```

[beatfactor]

And do you have any other relevant details? Like config, verbose logs etc? Please have a look at the contributing guidelines for submitting an issue.

This report isn't helpful for us to diagnose the problem.

[alesanu]

config:

    "theoryWin7CH69" : {
      "globals" : {"env" : "theory"},
      "launch_url" : url,
      "selenium_host" : "hub.browserstack.com",
      "selenium_port" : 80,
      "silent" : true,
      "disable_colors": false,
      "screenshots" : {
        "enabled" : false,
        "path" : ""
      },
      "desiredCapabilities" : {
        "project" : "theoryWin7CH69",
        "browserName" : "Chrome",
        "version" : "69.0",
        "os" : "WINDOWS",
        "os_version" : "7",
        "resolution" : "1920x1080",
        "javascriptEnabled" : true,
        "acceptSslCerts" : true,
        "browserstack.debug" : true,
        "browserstack.user" : "user",
        "browserstack.key" : "key"
      },
      "skip_testcases_on_fail" : false,
      "end_session_on_fail" : true
    },
=========================================================
"src_folders" : ["./tests"],
  "output_folder" : "./reports",
  "custom_commands_path" : "",
  "custom_assertions_path" : "./custom-assertions",
  "page_objects_path" : "./pages",
  "globals_path" : "../globals/global.js",
  "live_output" : false,
  "detailed_output": false,
  "parallel_process_delay" : 500,
  "disable_colors": true,```

[beatfactor]

Still not helpful. Have you got anything else?

[alesanu]

Same output using verbose logs. I'll paste the end of the log here.

   { using: 'css selector',
     value:
      'body > main > div > div > div.zone-content.gb-wrapper > div > div.project-list' }
   Response 200 POST http://hub-cloud.browserstack.com/wd/hub/session/e59ee1ec19794b45bec0572a4b81e10bfff4aab9/element/6/elements (33ms)
   { state: 'success',
     sessionId: 'e59ee1ec19794b45bec0572a4b81e10bfff4aab9',
     value: [ { ELEMENT: '36' } ],
     status: 0 }
   Request GET http://hub-cloud.browserstack.com /wd/hub/session/e59ee1ec19794b45bec0572a4b81e10bfff4aab9/element/36/displayed  
   Response 200 GET http://hub-cloud.browserstack.com/wd/hub/session/e59ee1ec19794b45bec0572a4b81e10bfff4aab9/element/36/displayed (45ms)
   { state: 'success',
     sessionId: 'e59ee1ec19794b45bec0572a4b81e10bfff4aab9',
     value: true,
     status: 0 }
 → Completed command waitForElementVisible ({name, __index, selector, locateStrategy, parent) (121ms)
 → Running [afterEach]:
 → Completed [afterEach].
✔ [Widgets/Timelines/Topics Timeline] Click on one topic with one level (35.919s)
 → Running [beforeEach]:
 → Completed [beforeEach].

<--- Last few GCs --->

[1535:0x3655780]  7068187 ms: Mark-sweep 1392.5 (1425.7) -> 1392.0 (1425.2) MB, 964.0 / 0.0 ms  (average mu = 0.081, current mu = 0.003) allocation failure scavenge might not succeed
[1535:0x3655780]  7068750 ms: Mark-sweep 1392.7 (1425.2) -> 1392.2 (1425.7) MB, 560.2 / 0.0 ms  (average mu = 0.051, current mu = 0.005) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x35a755dbe1d]
    1: StubFrame [pc: 0x35a755dd1df]
Security context: 0x12254e21e6e1 <JSObject>
    2: join [0x39ee00a38e79] [path.js:~1146] [pc=0x35a7589d3d6](this=0x39ee00a02201 <Object map = 0x575b51b6601>)
    3: arguments adaptor frame: 2->0
    4: loadModule [0x1c69fb313e81] [/var/lib/jenkins/workspace/functional-product-topics-theoryWin7CH69/node_modules/nightwatch/lib/api-loader/_base-loader.js:~150] [pc=0x35...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc1c0 node::Abort() [node]
 2: 0x8dc20c  [node]
 3: 0xad60ae v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad62e4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xec3972  [node]
 6: 0xec3a78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xecfb52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xed0484 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xed30f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe9c574 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x113beae v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x35a755dbe1d 
[07:10:54] Finished 'functional tests' after 1.96 h
Set build name.
New build name is '#844-topics'
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? ```

[alesanu]

Sometimes it runs without errors Same output with node@8.15 and @10.15

[andrewgill02]

I am also seeing this issue with our tests, but it isn't just a single test that fails - it fails after the entire suite appears to have successfully run.

<--- Last few GCs --->

[64:0x2a66f00] 518021 ms: Mark-sweep 724.3 (781.2) -> 724.2 (781.2) MB, 86.6 / 0.0 ms (average mu = 0.995, current mu = 0.000) allocation failure GC in old space requested [64:0x2a66f00] 518100 ms: Mark-sweep 724.2 (781.2) -> 724.2 (750.2) MB, 79.5 / 0.0 ms (average mu = 0.989, current mu = 0.001) last resort GC in old space requested [64:0x2a66f00] 518180 ms: Mark-sweep 724.2 (750.2) -> 724.2 (750.2) MB, 79.9 / 0.0 ms (average mu = 0.978, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x3934203cfc5d]

Security context: 0x1fc6d5d1d9b1 1: byteLength(aka byteLength) [0x19012999e549] [buffer.js:522] [bytecode=0x2a9609043ac9 offset=204](this=0x0f1317c825b1 ,0x37e2ac244b59 <Very long string[352091660]>,0x1fc6d5d7ead9 <String[4]: utf8>) 2: arguments adaptor frame: 3->2 3: fromString(aka fromString) [0x19012999e509] [buffer.js:333] [bytecode=0x2a960903e2c1 offset=77](this=...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 0x9590c0 node::Abort() [node] 2: 0x959d66 node::OnFatalError(char const, char const) [node] 3: 0xb39fae v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xb3a1e4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xf39832 [node] 6: 0xf4993f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node] 7: 0xf12b55 [node] 8: 0xf1a41b v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node] 9: 0x1067738 v8::internal::String::SlowFlatten(v8::internal::Isolate, v8::internal::Handle, v8::internal::PretenureFlag) [node] 10: 0xb3802b v8::internal::String::Flatten(v8::internal::Isolate, v8::internal::Handle, v8::internal::PretenureFlag) [node] 11: 0xb43d89 v8::String::Utf8Length(v8::Isolate*) const [node] 12: 0x9397a9 [node] 13: 0xbc370a [node] 14: 0xbc42b9 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object*, v8::internal::Isolate) [node] 15: 0x3934203cfc5d

We are using nightwatch 1.0.18

I can't really give a reproduction test case as it happens when the entire suite has ran. I managed to work around this issue by adding the command --max_old_space_size=2048 to node.

A lot of the tests make calls to External APIs to create and fetch data.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.