Heap out of memory on all but dev builds

[marcusjwhelan]

Any time I try to use electron-webpack for building or production testing I get a heap out of memory crash. It works fine if I do electron-webpack dev but that is the only case it works.

I am currently trying to make a win distribution with electron-builder. Do I need to make my own webpack files. I thought that electron-webpack would handle this for me, since electron-builder is recommended. Here is the heap error.

 93% [1] after seal                                                                          
<--- Last few GCs --->

[32727:0x3ab9970]    29606 ms: Mark-sweep 1371.1 (1424.3) -> 1370.7 (1425.3) MB, 450.1 / 0.0 ms  (average mu = 0.112, current mu = 0.008) allocation failure scavenge might not succeed
[32727:0x3ab9970]    30228 ms: Mark-sweep 1371.9 (1425.3) -> 1371.5 (1426.3) MB, 618.9 / 0.0 ms  (average mu = 0.056, current mu = 0.006) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x35bf91bdbe1d]
    1: StubFrame [pc: 0x35bf91bdd1df]
Security context: 0x329cd8a9e6e9 <JSObject>
    2: getIntersectionType(aka getIntersectionType) [0x2cf4bdc23249] [/home/marcus/work/ClientElectron/node_modules/typescript/lib/typescript.js:~39985] [pc=0x35bf9244c452](this=0x3f34c67026f1 <undefined>,types=0x3760a1b4de11 <JSArray[23]>,aliasSymbol=0x3f34c67026f1 <undefined>,aliasTypeArguments=0x3f34c67026f1 <undefine...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [node]
 2: 0x8dc55c  [node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xec7bf2  [node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x35bf91bdbe1d 

[marcusjwhelan]

I am using typescirpt as well. Node v10.15.3, Typescript 3.3.4, electron 4,

[loopmode]

I had similar issues trying to embed really large files - in my case it was 3d models that went into hundreds of MB in size. It worked fine in webpack-dev-server but it crashed the build process.

The solution was to increase the memory available to node using the --max-old-space-size argument.

I must admit that I don't remember the difference between the electron-builder and electron-webpack commands - it's been some 2 years since I delivered the project... But this is from the scripts section of its package.json:

    "clean": "rm -rf dist/*",
    "compile": "electron-webpack",
    "dist": "yarn clean && yarn dist:compile && yarn dist:build",
    "dist:compile": "cross-env NODE_ENV=production NODE_OPTIONS='--max-old-space-size=8192' yarn compile",
    "dist:build": "cross-env NODE_ENV=production NODE_OPTIONS='--max-old-space-size=8192' electron-builder",

We could successfully build the app using yarn dist.

[marcusjwhelan]

@loopmode for me I increased the size but that does not help. Letting may application open up and just sit on the page the memory will slowly creep to a heap overflow and kill typescript process. I think maybe at this point its something wrong with my typescript config along with IntelliJ doing something weird. I will post on Typescript about this.

[loopmode]

Okay, I have no experience with typescript in this context at all. To be honest, it sounds like a memory leak resulting from a programming error somewhere in your code. But of course it might be due to electron-webpack configuration. I hope you can find and fix it.

[marcusjwhelan]

@loopmode any advice for looking for a memory leak? I have never encountered one before.

[loopmode]

Hard to decide where to start :) It's a broad topic and definitely worth researching, e.g. "finding memory leaks with chrome devtools". For electron, there are probably some additional caveats, but the concepts are similar. Same goes for node.js itself - you'd debug and hunt memory leaks with chrome devtools as well.

I can't give good advice without asking many many questions or looking at the code/project. But I'll try.

Typically, a memory leak in javascript means that you loaded or created something, and you still keep a reference to it - so the garbage collector can't remove it from memory, even if the object itself isn't really needed anymore. Because somewhere, a variable exists and keeps the object as its value.

For example, imagine you have a reference to some DOM element, e.g. var myElement = document.getElementById('my-element');. Even if you remove that element from the DOM, e.g. myElement.parentNode.removeChild(myElement) - it won't be removed from memory until you reset the reference to it, e.g. by doing myElement = undefined or myELement = null. This goes also for all of its children etc.

Whenever you use addEventListener, do not forget to use removeEventListener again when you're "done" with the job. Careful when loading images programmatically and using e.g. addEventListener('load'). Careful when using intervals.

To be honest, I can't go deeper into this, and I didn't even really scratch the surface.. However, maybe some of these links can help you gain understanding: https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/ https://developers.google.com/web/tools/chrome-devtools/memory-problems/ http://seenaburns.com/debugging-electron-memory-usage/ https://marmelab.com/blog/2018/04/03/how-to-track-and-fix-memory-leak-with-nodejs.html

[marcusjwhelan]

@loopmode https://github.com/webpack/webpack/issues/8431 here is an issue that I had started on webpack. My question is how do i edit the ForkTSwebpackplugins default values?

[marcusjwhelan]

@loopmode I was able to get development working once I commented out any ForkTSwebpackplugins code in electron-webpack. Still crashing on compilation/build though. Gets to 93% seal then crashes. Upped the memory to my computers max. It gets to 93% then the memory slowly creeps up to max and crashes.

[loopmode]

If I understand correctly, without the ForkTSwebpackplugin, you get a bit farther, but then it still crashes. Thus, the problem is elsewhere and ForkTS stuff is "amplifying" it, right?

You could try to set up a fresh project based on electron-webpack-quick-start. It shouldn't suffer your problems right away, or other users would have reported the issue already. Then you could add dependencies and some of your code, step by step, and see whether it starts showing the symptoms at some point to pinpoint the culprit.

Or can you share the actual project? The idea would be to get a minimal reproducible example repo.

[marcusjwhelan]

@loopmode Well not for development. Once I removed the code for ForkTS development runs smoothly. It is compiling the code that then breaks. I do such with this project https://github.com/marcusjwhelan/testingproj, however this project works fine. That project though will start showing the same symptoms once I add more reducers and actions.

[marcusjwhelan]

@loopmode was able to compile without any issues once I went into /configurations/ts.js changed const isTranspileOnly to true and bam almost instance compilation. Tried setting up my own webpack file to add to electron-webpack but did not help. It seemed anything I put in there did not go through. I think all the TS checking is fine but with very large projects it fails.

[loopmode]

@marcusjwhelan according to https://github.com/electron-userland/electron-webpack/blob/master/packages/electron-webpack/src/configurators/ts.ts#L14 it should be enough to have electron-webpack-ts as a devDependency and set NODE_ENV to production - then transpileOnly would be true..

[marcusjwhelan]

@loopmode the electron-wepback-ts is fine to have, but using command cross-env NODE_ENV=production NODE_OPTIONS='--max-old-space-size=8192' electron-webpack it will fail. But if I change !configurator.isProduction to configurator.isProduction it works.

[marcusjwhelan]

Also though. I need isTranspileOnly to be true in development as well. Otherwise it crashes too.

[marcusjwhelan]

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.configureTypescript = void 0;

function _bluebirdLst() {
  const data = require("bluebird-lst");

  _bluebirdLst = function () {
    return data;
  };

  return data;
}

var path = _interopRequireWildcard(require("path"));

function _util() {
  const data = require("../util");

  _util = function () {
    return data;
  };

  return data;
}

function _interopRequireWildcard(obj) { if (obj && obj.__esModule) { return obj; } else { var newObj = {}; if (obj != null) { for (var key in obj) { if (Object.prototype.hasOwnProperty.call(obj, key)) { var desc = Object.defineProperty && Object.getOwnPropertyDescriptor ? Object.getOwnPropertyDescriptor(obj, key) : {}; if (desc.get || desc.set) { Object.defineProperty(newObj, key, desc); } else { newObj[key] = obj[key]; } } } } newObj.default = obj; return newObj; } }

let configureTypescript = (() => {
  var _ref = (0, _bluebirdLst().coroutine)(function* (configurator) {
    const hasTsChecker = configurator.hasDevDependency("electron-webpack-ts");

    if (!(hasTsChecker || configurator.hasDevDependency("ts-loader"))) {
      return;
    } // add after js

    configurator.extensions.splice(1, 0, ".ts", ".tsx");
    const isTranspileOnly = true
    const tsConfigFile = yield (0, _util().getFirstExistingFile)([path.join(configurator.sourceDir, "tsconfig.json"), path.join(configurator.projectDir, "tsconfig.json")], null); // check to produce clear error message if no tsconfig.json

    if (tsConfigFile == null) {
      throw new Error(`Please create tsconfig.json in the "${configurator.projectDir}":\n\n{\n  "extends": "./node_modules/electron-webpack/tsconfig-base.json"\n}\n\n`);
    }

    if (configurator.debug.enabled) {
      configurator.debug(`Using ${tsConfigFile}`);
    } // no sense to use fork-ts-checker-webpack-plugin for production build

/*

    if ((isTranspileOnly && !configurator.isTest)) {
      const ForkTsCheckerWebpackPlugin = require("fork-ts-checker-webpack-plugin");

      configurator.plugins.push(new ForkTsCheckerWebpackPlugin({
        tsconfig: tsConfigFile,
        logger: configurator.env.forkTsCheckerLogger || {
          info: () => {// ignore
          },
          warn: console.warn.bind(console),
          error: console.error.bind(console)
        }
      }));
    }
*/

    const tsLoaderOptions = {
      // use transpileOnly mode to speed-up compilation
      // in the test mode also, because checked during dev or production build
      transpileOnly: isTranspileOnly,
      appendTsSuffixTo: [/\.vue$/],
      configFile: tsConfigFile
    };

    if (configurator.debug.enabled) {
      configurator.debug(`ts-loader options: ${JSON.stringify(tsLoaderOptions, null, 2)}`);
    }

    configurator.rules.push({
      test: /\.tsx?$/,
      exclude: /node_modules/,
      use: [{
        loader: "ts-loader",
        options: tsLoaderOptions
      }]
    });
  });

  return function configureTypescript(_x) {
    return _ref.apply(this, arguments);
  };
})(); exports.configureTypescript = configureTypescript;
// __ts-babel@6.0.4
//# sourceMappingURL=ts.js.map

[marcusjwhelan]

I am curious to know what specifically it touched when running tests though. It seems that I can't run the simplest test now.