How to improve load times when using electron-compile?

[aeneasr]

I have a react app which uses electron-compile for jsx/es6. However, the inital loading of the BrowserWindow takes about 1-2 seconds on a fast system, and 5-10 seconds on slower ones. I believe that the root cause of this is slow require calls. So I asked myself if electron-compile actually bundles the app, or simply transpiles it and (probably?) compresses it.

Is it possible to compile a create-react-app like bundle instead of all the files? Would that increase loading times?

[aeneasr]

I set up a repo that uses electron-compile in one window, and in another it uses a create-react-app bundle (slightly modified).

The load time on the left is electron-compile, the one on the right is the cra bundle:

As you can see, the bundle needs about 50% less time than the electron-compile app. Both ran with NODE_ENV=production. I think this makes a strong case for bundling code that will run in the browser window.

[MarshallOfSound]

@arekkas Are you using an ASAR archive? It normally speeds these things up by quite a bit 👍

[Hum4n01d]

@MarshallOfSound What about in development?

[MarshallOfSound]

@Hum4n01d Bundling in development would result in much slower build times that you currently see, a second or so extra in development is not a big deal IMO as

1. It's development, no consumer is negatively impacted
2. It's a second or so... For the developers... 😆

Also key thing to realize here, electron-compile compiles on demand so the extra load time you see is also it compiling the files. The comparison the CRA is invalid as it is already bundled / compiled when you request it 👍

[anaisbetts]

@arekkas Are you in development or production mode? electron-compile operates much slower in the former case (but the advantage being, things get compiled incrementally). That being said, right now we are a fair bit slower than Webpack because of the node module system (600ms vs 1100ms startup time in the app I'm currently working on). I'm looking into ways to improve this.

[aeneasr]

I was running this in development. I'll bundle it today in production mode (asar) and report back on my findings.

[aeneasr]

@paulcbetts would you recommend electron-forge or electron-builder for a comparison?

[aeneasr]

I am very convinced that this is caused by the require calls that a webpack bundle avoids. Here are the runtimes using an app built with electron-forge package ând asar: true:

Again, left is electron-compile, right is bundle.

The load times get increasingly worse on slower systems:

@MarshallOfSound While I know how satisfying it is to close an issue, it should be reopened as there are significant load time differences not only in development, but also in production. A slowdown of about 3 fold on production builds and slow machines is considerable and deserves an issue.

Thank you all for participating on this and putting your work into it.

[aeneasr]

The issue gets much worse when I add imports that have a lot of npm dependencies. On slower systems the increase is very noticable. Here is the same app from above with a bunch of added npm dependencies. The screenshots are production, not development. In development, the increase is even worse, but as @MarshallOfSound noted it does not really matter if development builds are slower. It actually helps engineers understand what happens when your customer is using a slow machine :D

[MarshallOfSound]

@arekkas Can you provide the electron-compile sample app you are using to generate those numbers, will try this out on my own machine. The slowdown is unlikely to be electron-compile's fault rather the fault of the node module loader (I think).

[aeneasr]

@MarshallOfSound sure, it's here. To run it, do:

$ npm i
$ npm run start

To compile it:

$ npm run package

... the fault of the node module loader (I think).

I agree

The slowdown is unlikely to be electron-compile's fault rather ...

I disagree. It's not the fault of electron-compile explicitly but rather implicitly. The cause is that electron-compile does not bundle the files that are shown in the BrowserWindow. Yes, the root cause is that the module loader of electron is slow but unfortunately there are no fixes for that. It has been extensively discussed in the electron repository: https://github.com/electron/electron/issues/169

Thus I think we should come up with ways to improve electron-compile performance by looking into bundling BrowserWindows using e.g. webpack or react-scripts (CRA).

edit:// The repository is a bit noisy because I was using it to deal with other issues as well. Sorry about that.

[MarshallOfSound]

@arekkas I don't have a windows machine to trial this on right now but I ran it on my mac and the results are drastically different, the results you are seeing are probably due to the excessive calls to fs.lstatSync. When I apply the lstatSync patch that @paulcbetts wrote the electron-compile code actually loads faster than Create React App

See @paulcbetts gist https://gist.github.com/paulcbetts/da85dd246db944c32427d72026192b41

The patch I applied in index.html is this

const lru = require('lru-cache')({ max: 256, maxAge: 250 });
const fs = require('fs');
var origLstat = fs.lstatSync.bind(fs);
fs.lstatSync = function(p) {
    let r = lru.get(p);
    if (r) return r;

    r = origLstat(p);
    lru.set(p, r);
    return r;
};

@paulcbetts Are you aware of any negative affects caused by this caching or can we add it to https://github.com/electron/electron-compile/blob/master/src/es6-shim.js to get this speed boost to everyone?

[aeneasr]

@MarshallOfSound the electron-compile window is not showing the react app, maybe that's why it loads so fast

[MarshallOfSound]

@arekkas I did notice that, thought they were different 😆 Will see why things aren't loading. The repo is quite strange 😄

Regardless of the outcome here I think that bundling JS files is not something that -compile will do. It's a lot of logic and code to make a bundler and it causes a lot of issues with things like paths and relative dynamic files and other such magic that you expect to work in a node environment but breaks when you start bundling. This is all IMO, @paulcbetts might have some crazy cool ideas in this area or might disagree but I definitely think that bundling is way out of the realm electron-compile wants to venture in.

That said I am interested in solving this issue in any other way so I'll be exploring how to speed up the node module loader in my free time 👍

[aeneasr]

The repo is quite strange

I can set up a clean repository if you want to and if you think it will help. Let me know!

That said I am interested in solving this issue in any other way so I'll be exploring how to speed up the node module loader in my free time

That's great to hear! Thank you!

[aeneasr]

Unfortunately, I can not reproduce your findings on my MacBook Pro (Late 2013). This is in dev mode:

This is in prod mode:

As you can see, the difference is significant on a mac as well.

ps: I tried the repo with my instructions from above and it worked out of the box:

$ git checkout -b performance origin/performance
$ cd forge
$ npm i
$ npm run start
$ npm run package

edit:// fixed typo

[anaisbetts]

Can someone actually save the timeline file instead of just showing me pictures of it? :P

[aeneasr]

Not sure how that is possible in chrome. Any recommendations?

[anaisbetts]

@arekkas Right-click on the graph

[aeneasr]

Here you go

timelines.zip

[aeneasr]

Can we actually reopen this issue? I feel like this needs to be adressed and having this closed is kind of telling me: not gonna touch it.

By the way, webpack now supports electron targets for main process as well as render processes. With configurable entrypoints it should actually not be too hard getting this done, and then benefiting from huge performance gainzz: https://webpack.js.org/configuration/target/#target

[aeneasr]

.. especially because the reason for closing this issue is actually not resolving the issue, as shown in the comments that followed.

[anaisbetts]

@arekkas Agree

[anaisbetts]

@arekkas So, I had a look at your trace - in the electron-compile case, node-uuid is initializing itself which takes 1/3rd of the total time, where webpack doesn't do it. This kinda hints to me that node-uuid probably won't work or will initialize later? This is probably because in the electron-compile case, process.platform is real but in webpack, process is a stomped variable that only has process.env.

That being said, because you're also loading a ton of modules, you're also being punished by the fs.lstatSync stuff that we're talking about in this issue that is legit. If you upgrade to very latest Electron, it helps a lot with this, but it's definitely still A Thing.

[aeneasr]

Thanks for looking into this, I'll check out latest electron and also check if I can replace node-uuid

[anaisbetts]

At Slack we had to ditch node-uuid for this very reason too

[aeneasr]

Yeah I actually need pseudo random, not like super duper distributed unguessable magic here. I'll check it out, thanks so much for letting me know.

[aeneasr]

... on a side note, I think giving webpack + electron targets is worth a shot here! I don't really know how we would configure that, but I think it would greatly improve performance of electron apps.

[alexstrat]

Has anyone considered / tried to use v8 snapshots like Atom did?

[chgibb]

@arekkas Electron startup times is something I've been interested in for a few years. This piece of documentation from a project I'm involved in might be of help with respect to minimizing startup times.

[aguynamedben]

I've been using a Webpack approach, as guided by https://github.com/chentsulin/electron-react-boilerplate, and it "works", but the second I try to use @paulcbetts great electron-remote, which assumes pure Node.js behaviors for require.resolve(), and umzug which tries to load migrations dynamically by assuming pure Node.js behaviors for module.

Having Webpack stomp all over Node.js and losing access to these important dependencies is a major bummer. It appears there's still no "best practice" for moving CPU-heavy work to a 3rd process using Electron+Webpack, aside from DIY. I've done a DIY approach, spinning up an additional renderer process pointed at a Webpack bundle, having them talk over gRPC, but doing all that myself when there is a perfectly good dependency available is a bummer.

The complexity cost of adding Webpack is non-trivial. And I'm not a Webpack hater, I've gotten great use out of it in other projects, but Webpack+Electron seems like a bummer when you have great contributors @MarshallOfSound and @paulcbetts working finely without it.

[aguynamedben]

The V8 snapshots @alexstrat mentions are very interesting.

It seems like the goal of this issue could be refined to be a little bit more specific:

1. Increase load time for production build (more important, use startup snapshot approach, electron-link)
2. Increase load time for development (less important, lean harder on HMR? not sure)

[gmaclennan]

I have not tried this, but https://github.com/dominictarr/noderify might help with load times from many require calls.