🚀 perf: reduce memory usage by using less stats object

endiliey commented 5 years ago

🚀 Performance improvements (:rocket:)

Summary

Attempt to reduce memory usage #17 and increase build performance

Technically instead of doing

this.getAssets(this.stats.chunks);

We did

this.getAssets(this.getMinimalStatsChunks(compilation.chunks));

What this getMinimalStatsChunks did is basically mimicking webpack way of creating stats.chunks from https://github.com/webpack/webpack/blob/1a773336187ec0218e060d453f801bc73ab98a6e/lib/Stats.js#L633

Why so ? Because there are lot of unwanted fields in stats.chunks that causes JavaScript heap out of memory crash. There are lot of expensive operations like sorting that we actually dont need for this plugin

Why

Fix #17

Checklist

[x] Your code builds clean without any errors or warnings
[x] You are using approved terminology
[ ] You have added unit tests, if apply.

Test Plan

Test Scenario

git clone https://github.com/leoz/docusaurus_crash.git
cd docusaurus_crash/website/
yarn
yarn build

Before

The log looks like this

endiliey@LAPTOP-5K3AJ21I:/mnt/c/Users/endij/Desktop/Linux/test/docusaurus_crash/website$ yarn build
yarn run v1.12.3
$ docusaurus build
Creating an optimized production build...

Compiling

● Client █████████████████████████ additional chunk assets processing (90%)

● Server █████████████████████████ emitting (95%) CopyPlugin

<--- Last few GCs --->

[21275:0x2d42dd0]    35816 ms: Mark-sweep 2038.2 (2050.3) -> 2038.0 (2050.8) MB, 1302.5 / 0.0 ms  (average mu = 0.176, current mu = 0.109) allocation failure GC in old space requested
[21275:0x2d42dd0]    37125 ms: Mark-sweep 2038.7 (2050.8) -> 2038.5 (2051.3) MB, 1304.5 / 0.0 ms  (average mu = 0.087, current mu = 0.003) allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1a86142]
Security context: 0x2fffaff9a299 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x3efc61980139] [/mnt/c/Users/endij/Desktop/Linux/test/docusaurus_crash/website/node_modules/webpack/lib/Stats.js:~568] [pc=0x125336d69482](this=0x19e34c3804d1 <undefined>,0x35d8dd2184d9 <ModuleReason map = 0x3019b2906769>)
    2: map [0x2fffaff8c2c9](this=0x0744b2da9c31 <JSArray[144652]>,0x3efc61980139 <JSFunction (sf...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191012.144558.21275.0.001.json
Node.js report completed
 1: 0x98c680 node::Abort() [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 2: 0x98d5e6 node::OnFatalError(char const*, char const*) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 3: 0xb077ce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 4: 0xb07b49 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 5: 0xf12c05  [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 6: 0xf1d56b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 7: 0xf1e287 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 8: 0xf20d25 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
 9: 0xeebd50 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
10: 0x11bdcbf v8::internal::Runtime_AllocateInTargetSpace(int, unsigned long*, v8::internal::Isolate*) [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
11: 0x1a86142  [/home/endiliey/.nvm/versions/node/v12.2.0/bin/node]
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command

After (patching the react-loadable-ssr-addon with this PR)

This repo test

endiliey commented 5 years ago

@themgoncalves what do you think of this PR ?

themgoncalves commented 5 years ago

@endiliey thanks for your contribution! I will be running some tests and getting back with the results.

💪

themgoncalves commented 5 years ago

@endiliey FYI v0.2.0 was released.

endiliey commented 5 years ago

Thanks. I'm gonna test it out. If any problem, I'll report back.

themgoncalves / react-loadable-ssr-addon