Plugin forking threads for each chunk

[djmetzle]

We're seeing a threading and concurrency problem trying to upgrade our CC webpack plugin.

We've moved from the old v3 version: https://www.npmjs.com/package/webpack-closure-compiler

To the latest v4 plugin: https://www.npmjs.com/package/closure-webpack-plugin

We're seeing a pretty wild performance degradation here. The previous v3 version used a tuneable concurrency flag to limit the number of parallel compilations performed by the plugin: https://github.com/roman01la/webpack-closure-compiler/blob/3677e5e6672af68eb4608491d39d9e39ba8bf55e/index.js#L99

This would use async.queue to only execute CC at a specified concurrency.

Testing the new plugin with the default configuration from the docs we see a separate thread spun for every chunk in our webpack compilation. We saw thousands of java threads, and knocked down our development environments.

For additional detail, we tested some different combinations of options. Using any of the platform options does not affect the result. We can use platform: javascript and then still see hundreds of node threads, as opposed to JVM threads.

We also tested switching the mode to AGRESSIVE_BUNDLE. When we run in AGRESSIVE mode though, we do not see more than one compilation at a time. Our compilation is not compatible with that option, but it is worth noting that the problem does not happen when using the AGRESSIVE option.

---

On investigation, it seems like the cause of this explosion in threads might be a bug. The code bifurcates on the bundling type: https://github.com/webpack-contrib/closure-webpack-plugin/blob/1246f6e51049a5d0afcd3b6ef2ef32a5bf3f9940/src/closure-compiler-plugin.js#L448-L452

In the standard codepath, there is a forEach over each chunk in the compilation: https://github.com/webpack-contrib/closure-webpack-plugin/blob/1246f6e51049a5d0afcd3b6ef2ef32a5bf3f9940/src/closure-compiler-plugin.js#L467 which pushes the chunk onto a compilations list: https://github.com/webpack-contrib/closure-webpack-plugin/blob/1246f6e51049a5d0afcd3b6ef2ef32a5bf3f9940/src/closure-compiler-plugin.js#L497-L498 And then Promise.alls them: https://github.com/webpack-contrib/closure-webpack-plugin/blob/1246f6e51049a5d0afcd3b6ef2ef32a5bf3f9940/src/closure-compiler-plugin.js#L553

But the AGRESSIVE code path only does a single compilation promise: https://github.com/webpack-contrib/closure-webpack-plugin/blob/1246f6e51049a5d0afcd3b6ef2ef32a5bf3f9940/src/closure-compiler-plugin.js#L776-L777

We suspect this explains the explosion in background work, and the difference in behavior between STANDARD and AGRESSIVE.

---

We're running webpack v4 on node 12 with an otherwise uninteresting configuration. 270 entrypoints, 2500 modules, and about 300 emitted chunks.

[ChadKillingsworth]

In AGRESSIVE mode, all of the chunks are passed to the compiler in one single compilation. The dependency tree is normalized and relationships are clearly defined.

In STANDARD mode, each entry point equates to a compilation. In this mode, webpack can duplicate a module into multiple output chunks. Closure Compiler doesn't support this so each chunk tree be compiled individually.

Your use case sounds like it's pushing limits here. What would be needed is a thread manager to limit the number of parallel compilations.

[djmetzle]

@ChadKillingsworth Thanks for the reply!

Your use case sounds like it's pushing limits here. What would be needed is a thread manager to limit the number of parallel compilations.

We weren't having problems with the v3 plugin though. The previous v3 version used a tuneable concurrency flag to limit the number of parallel compilations performed by the plugin.

I think that we need something like async.queue, or some other approach, like a "thread manager" to limit parallel work.