Very high memory usage (~10GB for ~3000 files)

[bew]

When the compilation uses A LOT of files, the ManifestPlugin uses A LOT of memory.

As an example, our compilation has ~2800 files (only assets: images, videos, documents), the ManifestPlugin used between 8 & 10GB of RAM to complete. This is way too much!

I've tracked the issue to be at line: https://github.com/danethurber/webpack-manifest-plugin/blob/bcca8907df46ad6b57199d34d7be876f95170a52/lib/plugin.js#L58 And more specifically, the toJson() call.

This method is used to get easy-to-use object with information about the webpack compilation. The plugin only uses the assets field of the generated object, here: https://github.com/danethurber/webpack-manifest-plugin/blob/bcca8907df46ad6b57199d34d7be876f95170a52/lib/plugin.js#L88

The very high memory usage happens while generating the fields related to modules, an easy fix is to disable this part:

-var stats = compilation.getStats().toJson();
+var stats = compilation.getStats().toJson({
+  modules: false,
+});

It'll still generate other data that the plugin doesn't use. Another fix is to disable everything we don't need:

-var stats = compilation.getStats().toJson();
+var stats = compilation.getStats().toJson({
+  version:     false,
+  hash:        false,
+  timings:     false,
+  builtAt:     false,
+  env:         false,
+  publicPath:  false,
+  outputPath:  false,
+  entrypoints: false,
+  chunks:      false,
+  chunkGroups: false,
+  children:    false,
+  modules:     false, // the memory hungry part is disabled here
+
+  assets: true, // only get assets information
+});

These two diffs resolves the problem, and the compilation uses normal amount of memory.

Let me know what you think about that

[Kenneth-KT]

Not sure if the author still want to maintain this library.

If not, there is another plugin webpack-assets-manifest that do exactly the same thing and they have this problem fixed in this commit already, but they haven't released it yet.

Run yarn add --dev https://github.com/webdeveric/webpack-assets-manifest.git\#051243e to add it to your repo.

[mastilver]

fixed in b8db85e

[bew]

Thanks @mastilver !