multiple html outputs webpack rebuild still slow

[asiainfoliwei]

Hello, First, The plugin always so nice,still I use it in a multiple html outputs project, Here is my version And , when i change a file, It takes a lot of time to rebuild, like it

[jantimon]

Can you please try to profile it?

I have created a plugin which helps you to generate a CPU profile:

https://github.com/jantimon/cpuprofile-webpack-plugin

[ar7casper]

@jantimon I'm having difficulties as well. I got ~25 entries with an HTML per entry.

I tried two approaches - single & multi webpack config (this & this) and both of them crash. In addition, tried running as webpack-dev-server & webpack-dev-middleware.

And the build itself is taking forever. Oh, and tried ver.4 as well, no success.

That's what I get eventually -

<--- Last few GCs --->

[97512:0x103000000]   403855 ms: Mark-sweep 1369.0 (1477.9) -> 1369.0 (1479.4) MB, 935.0 / 0.0 ms  allocation failure GC in old space requested
[97512:0x103000000]   404780 ms: Mark-sweep 1369.0 (1479.4) -> 1368.9 (1448.4) MB, 924.3 / 0.0 ms  last resort GC in old space requested
[97512:0x103000000]   405718 ms: Mark-sweep 1368.9 (1448.4) -> 1368.9 (1448.4) MB, 937.9 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x15d7de9a55e9 <JSObject>
    1: set(this=0x15d7b26fc4c1 <Map map = 0x15d757d848d9>,0x15d7b1acaff9 <String[66]: /********/node_modules/@material-ui/icons/Rowing.js>,0x15d7ea3a3899 <String[43]: ./node_modules/@material-ui/icons/Rowing.js>)
    2: shorten [/********/node_modules/webpack/lib/RequestShortener.js:~56] [pc=0x258b2ac6ad20](this=0x15d77626e639 <RequestShortener map = 0x15d7962fecc9>...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [*********/node_modules/.bin/node]
 2: node::FatalTryCatch::~FatalTryCatch() [/*********/node_modules/.bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/*********/node_modules/.bin/node]
 4: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/*********/node_modules/.bin/node]
 5: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/*********/node_modules/.bin/node]
 6: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [/*********/node_modules/.bin/node]
 7: 0x258b29a842fd
[1]    97511 abort      npm run dev

[jantimon]

Thanks @ar7casper could you try to attach a profile?

[ar7casper]

@jantimon Archive.zip

[ar7casper]

@jantimon Well, it got more interesting now -
I'm running it in two ways, A - dev-server, B - dev-middleware.

when running in A, I get the fail. when running in B, with eslint-loader it takes several minutes, w/o it, 35 seconds.

[ar7casper]

@jantimon running node --max-old-space-size=8192 server works. when working with webpack-dev-server it fails. the config is common between the two envs.

[jantimon]

From your profile:

Could you please try if removing eslint helps?

[zaaack]

@asiainfoliwei I hit this problem too, don't know why, but I wrote a simple html webpack plugin solved my issue, it's almost 20x faster then before, rebuilds only take <= 1s, here is: https://github.com/zaaack/htmls-webpack-plugin

[jantimon]

@zaaack the problem with htmls-webpack-plugin is that it doesn't support the webpack ecosystem. You can't use any loader for your templates.

[zaaack]

@jantimon Well, I'm not sure what's the common use cases of html webpack plugins, but it doesn't look like a problem for me, since most of the assets are bundled in jses, what I only need is generating simple html entrences for my spas...

[asiainfoliwei]

@asiainfoliwei I hit this problem too, don't know why, but I wrote a simple html webpack plugin solved my issue, it's almost 20x faster then before, rebuilds only take <= 1s, here is: https://github.com/zaaack/htmls-webpack-plugin

Hello zaaack, In my project, I use the ejs-loader as my template, So it is not suitable for me, Also thanks

[zaaack]

@asiainfoliwei Hi, asiainfoliwei. htmls-webpack-plugin also use ejs as default temple engine, but you can define a custom promise based template engine via render option. No offence, just curious what's the benefits of using loaders instead of a simple customized render function, since we can even call loaders directly in functions.

[asiainfoliwei]

Can you please try to profile it?

I have created a plugin which helps you to generate a CPU profile:

https://github.com/jantimon/cpuprofile-webpack-plugin

Hello jantimon, I use the plugin for Analysising Packaging time consuming, but it throws a error: TypeError: Cannot read property 'watchRun' of undefined at CpuProfileWebpackPlugin.apply (/Users/liwei/Desktop/dding/ydaz-dev/node_modules/_cpuprofile-webpack-plugin@1.10.1@cpuprofile-webpack-plugin/index.js:6:20)

[ar7casper]

@zaaack looks . nice, gonna check it out.

[ar7casper]

@jantimon Eventually, I did 2 things -

1. use webpack's Multi config (array of configs),
2. For the build, I've added parallel-webpack to run it.

[npm-ued]

Any other good solutions？

[jantimon]

@asiainfoliwei this works only for webpack 4+ @ar7casper did you try to remove eslint and create a new cpu profile? @zaaack we tried to offer the best compatibility to all stacks and plugins so we fully integrate with the webpack compiler. this allows loaders like https://github.com/GoogleChromeLabs/prerender-loader to work

There was also a very recent improvement which improved memory consumption a lot: https://github.com/jantimon/html-webpack-plugin/issues/1186

[ar7casper]

@jantimon

The linter was responsible for about ~60% of the time. Removing it helped a lot. The linting is now done in a CI process, making sure nothing gets merged without lint passing. The build itself is done without linter. When developing, you can pass a flag to build only a certain entry(ies).

Having said that, I'll probably give parcel a try, cause I hit a scale brick wall again

[zhangzippo]

Can you please try to profile it?

I have created a plugin which helps you to generate a CPU profile:

https://github.com/jantimon/cpuprofile-webpack-plugin hello,i have the same problem when i use html-webpack-plugin for 15 pages build, and i found the most of time waste for the sleep, there is the profle, could you tell me what the sleep means?

归档.zip

[jantimon]

Multi threading e.g. here it starts the minification for terser in multiple threads:

It looks like your main time consumer are the vue-loader and the babel-loader so I guess this is unrelated to the html-webpack-plugin

[Arech-github]

So, what's the solution to this problem? The only thing I can be sure of right now is that it will take me about 4 seconds to use this plugin, because I have about 40 pages, and if I comment out this plugin it will be built in less than a second

[jantimon]

@851201344 thanks for your feedback - can you please help us out and send a profile using the cpuprofile-webpack-plugin of an incremental build?

[Arech-github]

The webpack I used 4.35.0 html-webpack-plugin is 3.2.0 but I used cpuprofile-webpack-plugin , And there's no output

I saw the https://github.com/jantimon/html-webpack-plugin/issues/724

you said " Give it a try: https://www.npmjs.com/package/html-webpack-plugin/v/4.0.0-alpha.2 "

I tried it

but Error after use

and use this plugs it can output profiles

profiles.zip

It's really weird

Next I'm going to try out the parallel-webpack

Some of my other configuration files

Looking forward to your feedback . thanks

[jantimon]

The performance improvements are not included in 3.2 but only for 4.x

[Arech-github]

with 4.x @jantimon

[jantimon]

@851201344 sorry that is unrelated - your html-webpack-custominject-plugin is not compatible with the html-webpack-plugin 4.x

[Arech-github]

I fixed the bug and used the 4.x release and found that the build speed improved significantly

[Arech-github]

html-webpack-plugin 3.2.0

I just tried to comment out the AutoDllPlugin(autodll-webpack-plugin)

I noticed a build speed increase of about 5 seconds

If you encounter the same problem you can try what I did

[StickyCube]

I have 6 instances of HtmlWebpackPlugin in my config. I ran webpack devserver with --inspect to see where the bottlenecks were. Here's a sample from a single rebuild.

About half the time is spent in stats.toJson() which is called 6 times - one for each instance's templateParameters.

After setting templateParameters: false here is the profile again:

This halved my rebuild time.

Maybe something could be done to cache and share the return value from stats.toJson() between plugin instances?

[jantimon]

@StickyCube did you use html-webpack-plugin 4.x?

[StickyCube]

@jantimon I am using the following:

webpack@4.40.2 webpack-dev-server@3.8.0 html-webpack-plugin@3.2.0

Which I think are the latest versions of everything. Is 4.x still in beta? I could try install and see if it improves anything.

[jantimon]

Yes 4.x is in beta but already used by create-react-app. Did it improve the performance for you?

[VladimirPal]

I've faced the same issue. autodll-webpack-plugin - use html webpack plugin to inject dll dependecies. Dll plugin caches stats in stats.json file, but stats contain too much info, including sources, so it may weight really lot. For every html file the plugin will read and parse this file which is slow.

https://github.com/asfktz/autodll-webpack-plugin/issues/143

[jantimon]

Version 4.x will not use those stats anymore

[asanchezaguirre]

Hello @jantimon !! The 4.x version significantly modified the time, but I'm worried that it´s beta. Is there a probable date for it to be stable? Thanks

[Javiani]

Hey guys,

I was facing the same issue regarding performance when you have several html outputs using html-webpack-plugin, the ^4.0.0-beta.11 version has a huge performance improvement!!! Awesome job!!! Thank you!

Before updating the version, I was trying to change my webpack in order to serve pug with the express instance provided by devServer, once pug runs on node there's no need to compile all the files on every change, and it would be a lot fast than any plugin.

Then I realize that it's impossible to write the same code for webpack and node express version...

[jantimon]

Finally resolved as stable version:

https://dev.to/jantimon/html-webpack-plugin-4-has-been-released-125d

[ruoru]

any way in webpack 3？