JS heap out of memory when bundling

[ZoolWay]

Hi! I am trying to bundle the components of my application but SystemJS Builder runs quite long and then I get FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory.

I do not think my project is huge but all js, html, json and css files together make up 612 files (collected by globby).

I am using JSPM 0.17.0-beta.44 with SystemJS 0.20.18, SystemJS Builder 0.16.10, Plugin CSS 0.1.35, Plugin JSON 0.3.0 and Plugin Text 0.0.11. No transpilation by SystemJS involved (my gulp task utilizes typescript and SystemJS just reads the output js files), the JS code is ES6 in AMD format.

Complete error:

<--- Last few GCs --->

[24416:00000227427B79B0]   201807 ms: Mark-sweep 1403.9 (1450.3) -> 1403.9 (1450.3) MB, 3469.0 / 0.0 ms  last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000016FA9529891 <JS Object>
    1: node [D:\Working\project1\node_modules\babel-traverse\lib\index.js:~94] [pc=0000019B61D44262](this=0000031E22442839 <JS Function traverse (SharedFunctionInfo 000002341E9FB159)>,node=00000300C60A0C41 <a Node with map 000001D216CA4331>,opts=000003089A456D29 <an Object with map 000001D216CA4909>,scope=0000001354D6FC09 <a Scope with map 000001D216CA2C81>,state=...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The security context is not always the same, here is another variant:

<--- Last few GCs --->

[10620:0000021A7F21F500]   338119 ms: Mark-sweep 1404.0 (1464.2) -> 1404.0 (1464.2) MB, 2894.8 / 0.0 ms  allocation failure GC in old space requested
[10620:0000021A7F21F500]   341011 ms: Mark-sweep 1404.0 (1464.2) -> 1404.0 (1455.7) MB, 2891.8 / 0.0 ms  last resort
[10620:0000021A7F21F500]   343887 ms: Mark-sweep 1404.0 (1455.7) -> 1404.0 (1455.7) MB, 2875.9 / 0.0 ms  last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000002BFF49A9891 <JS Object>
    1: new constructor(aka Binding) [D:\Working\project1\node_modules\babel-traverse\lib\scope\binding.js:~12] [pc=000000451836CA58](this=00000078B2A1EAE1 <a Binding with map 000002226D865BF9>,_ref=00000078B2A1EAA1 <an Object with map 000002226D8666A1>)
    3: registerBinding [D:\Working\project1\node_modules\babel-traverse\lib\scope\i...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[aluanhaddad]

Since you're using JSPM, does this work if you run build commands from the CLI? This might help track down the issue.

$ jspm build app --minify --format iife 

Also, strongly consider configuring TypeScript to output either system, es2015, or esnext modules rather than AMD modules.

Note that when using JSPM, rather than consuming the SystemJSBuilder API directly, you should instead acquire a pre-configured builder as in

var Builder = require('jspm').Builder;

var builder = new Builder();

This results in a SystemJSBuilder instance pre-configured with JSPM config that can be further customized as needed.

[ZoolWay]

Hm, I have not tried using JSPM CLI to build the app but in very short, it does not work.

Changing the module format to es2015 or system did not change anything, still getting JavaScript heap out of memory, same goes with obtaining the builder from jspm insead of using it directly.

I am preparting a minimum repository to share.

[ZoolWay]

Okay, I found out a bit more. I am now trying to bundle using JSPM like this:

jspm build dist/main.js app-bundle.js

When I removed the import for Kendo UI (import 'kendo.all.min';) from my files bundling started to work without heap out of memory.

I tried to reproduce this in a minimal example using the free OpenSource variant of Kendo UI but there the issue does not seem to happen.

Then I tried to exclude Kendo UI from the main bundle but that does not work either.

jspm build dist/main.js - kendo.all.min app-bundle.js

It also produces heap out of memory. Also tried kendo.all.min.js.

Any ideas how to solve this?

[aluanhaddad]

What happens if you run

$ jspm build kendo.all.min.js

?

[ZoolWay]

Another update: Bundling with Kendo UI works if I use --skip-source-maps...

[ZoolWay]

Running jspm build js/kendo/kendo.all.min.js (path is provided in map parameter otherwise) I still get heap out of memory. Adding --skip-source-maps works:

     Creating the single-file build for js/kendo/kendo.all.min.js...

       js/kendo/kendo.all.min.js

ok   Built into test.js, unminified as umd.

[ZoolWay]

I must add that deactivating the sourcemaps looks like only a temporary workaround. After the project got a little bigger I started to run into the heap out of memory issue again, especially when multiple gulp tasks are chained. I can work around it by increasing the NodeJS memory limit (i.e. --max_old_space_size=4096 gives it 4GB instead of 1.76GB) but I guess bundling 200MB if JS files should work using 1GB of RAM? Maybe there is some kind of memory leak?

[aluanhaddad]

@ZoolWay thank you for keeping this up to date and reporting your findings.

If you are running gulp tasks in a single process such that multiple bundling operations are potentially interleaved, you may indeed run into issues.

[ZoolWay]

What is the recommended process around this (although giving it just 4GB RAM might be a normal solution too)? Can in series running gulp tasks be forked into own processes?

[aluanhaddad]

Yes you can fork them, but running them sequentially should also do the trick. Consider using async/await in your gulpfile.js to orchestrate sequential bundling tasks in a simple manner or just using promises. The heapsize idea is also good