AssertionError: You must launch a compilation thread before attempting to use it

rob-mccann commented 7 years ago

Hey - we're seeing this appear intermittently when using happypack.

ERROR in ./components/ad-preview/ad-preview.scss
    Module build failed: ModuleBuildError: Module build failed: AssertionError: You must launch a compilation thread before attempting to use it!!!
        at Object.compile (/myproj/fe/node_modules/happypack/lib/HappyThread.js:138:7)
        at Object.compile (/myproj/fe/node_modules/happypack/lib/HappyThreadPool.js:82:14)
        at HappyPlugin.compile (/myproj/fe/node_modules/happypack/lib/HappyPlugin.js:192:14)
        at Object.HappyLoader (/myproj/fe/node_modules/happypack/lib/HappyLoader.js:31:15)
        at runLoaders (/myproj/fe/node_modules/webpack/lib/NormalModule.js:195:19)
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:364:11
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:230:18
        at runSyncOrAsync (/myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:143:3)
        at iterateNormalLoaders (/myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:202:4
        at /myproj/fe/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:70:14
        at _combinedTickCallback (internal/process/next_tick.js:73:7)
        at process._tickCallback (internal/process/next_tick.js:104:9)

Webpack 3.6.0 Happypack 4.0.0

Our webpack config is a little too complex to share here, I'll update this when I have a condensed version.

I think this happens because we share a threadpool for multiple instances of webpack - I'm not sure if there's a nice way to work around it without spawning a new threadpool for each instance (and incurring the performance hit therein). I wonder if we could tell the threadpool to wait for X 'done' events before terminating the pool?

Current theory scratchpad

running webpack in parallel
~~SIGINT taking time to terminate the thread~~
we transpile our webpack config; perhaps that's outputting something incorrect

amireh commented 7 years ago

Is it possible for you to stop the threadpool manually? As in, do you have the ability to tell when all your builds are finished?

If the answer is yes, we can add an option to simply not have happypack stop the pool by itself (in case it was shared.)

rob-mccann commented 7 years ago

That sounds like it might work - we can detect when it's done via webpacks callback.

amireh commented 7 years ago

Hmm, I just checked the code and it seems that we're doing reference-counting and only really stopping the threadpool once all active compilers have emitted the "done" event. This makes things more complicated, the fact that you're seeing this error.

Is there any chance you're using a plugin that is doing something finicky after the compiler is done? The error seems to indicate it's coming from a SASS loader, but I can't see how that could happen.

rob-mccann commented 7 years ago

Would you mind linking to where you see that?

amireh commented 7 years ago

First, HappyPlugin invokes threadPool.stop when the compiler emits "done" (are you using --bail?) in https://github.com/amireh/happypack/blob/master/lib/HappyPlugin.js#L107
Then, HappyThreadPool removes that compiler[1] from the list of active compilers in https://github.com/amireh/happypack/blob/master/lib/HappyThreadPool.js#L90
Then, finally, if no active compilers remain, a condition tested in https://github.com/amireh/happypack/blob/master/lib/HappyRPCHandler.js#L19, the threadpool is stopped: https://github.com/amireh/happypack/blob/master/lib/HappyThreadPool.js#L93.

[1] that also is kept by a reference-counting mechanism, so the compiler will only be removed when all the plugins for that compilation invoke stop

rob-mccann commented 7 years ago

Just thinking aloud below and trying to get a grasp from reading code by sight, apologies, but perhaps this flow will reproduce the issue?

const config = {}; // this has loads of stuff in it; cut it for now

const threadPool = HappyPack.ThreadPool();

const getConfig = () => (
  merge({
    plugins: [
      new HappyPack({
        id: 'babel',
        threadPool: threadPool,
        loaders: [ 'babel-loader' ]
      }),
    ],
  }, config);
);

await webpack(getConfig());
await webpack(getConfig());

I'll see if I can get a test to fail.

rob-mccann commented 7 years ago

I've done some investigation:

Array of configs

webpack([ { ...configA }, { ...configB }]).run(done)

Running webpack with an array of configurations work as you describe; the compilers are counted in and out again.

Sequential

webpack(configA);
compiler.run(() => { webpack(configB).run() })

Running webpack one after eachother causes the threadPool to terminate the threads after the first webpack and start them up again before the second. Whilst it's not ideal behaviour (performance-wise), it shouldn't error.

Parallel

webpack(configA).run();
webpack(configB).run();

Running initiating two compiler.run, without waiting for one to complete will choose one of the above behaviours, depending on where the thread is at.

Other ideas

Stepping through the code, I don't see how it's possible for us to get to the state where the above error shows. I'll keep digging.

Things I've tried already to get the test to fail:

delaying the SIGINT using setTimeout; no effect.

amireh commented 7 years ago

Thanks for the great report.

Regarding the sequential and parallel examples you're providing - is there a legit use-case for those? Why would you do that over the first approach, feeding webpack with multiple configurations?

I believe that for those second and third cases it may actually be the best bet to provide you with a hook to opt-out of the thread pool shutting itself down, then you'll be responsible for doing that (once you know all compilations are done.)

rob-mccann commented 7 years ago

We have several, separate gulp tasks running doing various things in webpack. I'll see if we can somehow use gulp to compose a webpack config and then run it once. My hunch is that's not feasible, though.

I'm not sure if it's shutting down early or if open is not being called; can't yet reproduce the issue reliably :( Will keep investigating.

rob-mccann commented 7 years ago

I've added an updated list of theories to the opening post.

fancy-rabbit commented 4 years ago

same problem. invoke multiple compiler or pass multiple configs to compiler reproduces this.

amireh / happypack