HappyPack stalls build in a docker environment

[jfairley]

This was 100% configuration issue on my part, but it took a while for me to figure out. Maybe this issue will help someone else in the future...

I have builds running in Docker (with CircleCI). According to CircleCI, the container has 2CPU/4096MB available. In practice, I've seen builds stall once a node process consumes over ~1300MB. I already knew this was an issue with my Jest tests due to a different Jest / zone.js issue, but yesterday my webpack build started stalling, too.

I'm using a HappyPack thread pool as follows:

HappyPack.ThreadPool({ size: require('os').cpus().length - 1 })

Eventually I noticed:

Well yeah; that's the problem. This is the part where I make a guess, but clearly require('os').cpus() doesn't give the expected data inside a Docker container.

I ended up hard-coding the thread pool size when in CI environment.

HappyPack.ThreadPool({ size: process.env.CI ? 3 : require('os').cpus().length - 1 });

HappyPack is happy again. 🍻

[jfairley]

@amireh Of course, nothing to do here. Close this issue when you're ready. It's merely here as documentation.

[amireh]

Ouch. Thanks for sharing that!

This is the part where I make a guess, but clearly require('os').cpus() doesn't give the expected data inside a Docker container.

Personally, I wouldn't even do this even on the host machine. Upon profiling several projects on two different machines with different specs and operating systems, I've found the "happy" value to range between 3 to 5 threads (assuming the host has >= 5 virtual threads, of course.)

In theory, assigning more workers could make it faster but only if the files are large enough that the overhead of IPC is justified, but I've yet to come across a project where that actually holds true.

The "threads" option documentation briefly touches on this topic. It's stated lightly because I have no empirical evidence as it is to back any of this up. 😁

It may be helpful to add a note of avoiding the use of os.cpus() in the README.

[jfairley]

BTW, I updated my config to remove the thread pool altogether. I really didn't need it, and as I read the documentation more closely, I saw that it's useful for when one has multiple different usages of HappyPack in the same build. I don't; I was only using it for the ts-loader.

Removing the thread pool, and thereby using the default of 3 threads, reduced my build times to about 25% of the time it took with n-1 threads.

[amireh]

Thanks again for sharing the information, will close now.