Try to move LanguageService to separate process

timocov commented 6 years ago

I noticed that ts-loader runs TypeScript synchronously in the main process what causes blocking webpack's main process and webpack stops doing other stuff with other files until TypeScript finish compiling (I guess it is applicable for projects with mixed typescript and javascript files).

It is possible that we can move LanguageService to other process and handle compilation there while webpack doing other work in the main process. In this case, I believe, we will compile in other process while webpack doing own work in the main process without blocking (but yeah, at some time we'll wait until the compilation is finished because there are no other non-typescript files to process by webpack).

(moved from https://github.com/TypeStrong/ts-loader/issues/825#issuecomment-416895826)

johnnyreilly commented 6 years ago

Feel free to spike this and send a PR to collaborate on. Could be interesting!

felixfbecker commented 6 years ago

Doesn’t this work with thread-loader?

johnnyreilly commented 6 years ago

@felixfbecker also my original thought; there is a distinction - see discussion in linked thread. This is the relevant quote from @timocov:

Not exactly. We can use fork-ts-checker-webpack-plugin along with transpileOnly: true flag to speedup performance, right? In some cases we can't use transpileOnly flag (for example if we use global const enums to inline some common constants).

timocov commented 6 years ago

Doesn’t this work with thread-loader?

If I understand correctly how thread-loader works, I do not think that it may help us. If we'll use thread-loader - we'll have some processes with ts-loader in each one and we still may block main process.

felixfbecker commented 6 years ago

thread-loader is not the same as fork-ts-checker-webpack-plugin: https://github.com/webpack-contrib/thread-loader https://github.com/Realytics/fork-ts-checker-webpack-plugin

We use thread-loader with ts-loader together. @timocov could you clarify what part of ts-loader still blocks the main process with that setup?

timocov commented 6 years ago

@felixfbecker I cannot tell you for sure, but compilation TypeScript is not async and every time when you request to compile some file it blocks the process until compilation is finished.

In our case we have the following situation:

webpack starts bundling
progress shows something like 12% building modules 23/55 modules 32 active PATH
then, when webpack meets the first TypeScript file, the progress stops for some time (every time when I observe this waiting it was on file with *.ts extension - also profiling shows the same)
in our case after 1-2 minutes webpack continues process files and progress continue too
every time when webpack meet TypeScript file it blocks for some time again, but only the first block is such long (the second and further blocks takes much less, but still it is noticeable time)

felixfbecker commented 6 years ago

Yes, but isn't the point of thread-loader that it runs the whole loader (i.e. ts-loader) in a completely separate process? So how would ts-loader then still be able to block the main process?

Have you tried using thread-loader in your setup?

timocov commented 6 years ago

Hm, good point - I didn't think about it.

So how would ts-loader then still be able to block the main process?

I need to check it. Quite possible that it can help here. 👍

timocov commented 6 years ago

I just have played with thread-loader and here are results:

It seems that it does the same thing I suggest to do in ts-loader. Thanks @felixfbecker!
After compilation we have a lot of errors like Module build failed: TypeError: Cannot read property 'plugin' of undefined - I guess they are related to thread-loader's limitation Loaders cannot use custom loader API (i. e. by plugins).. @felixfbecker do you have such errors in your build?
Bundling time (even with errors) decreased down from 550s to 480s. <- @johnnyreilly

felixfbecker commented 6 years ago

No, we don't have these errors. Here are the options we use for ts-loader, maybe this helps?

loader: 'ts-loader',
                        options: {
                            compilerOptions: {
                                target: 'es6',
                                module: 'esnext',
                                noEmit: false,
                            },
                            experimentalWatchApi: true,
                            happyPackMode: true, // typecheck in fork-ts-checker-webpack-plugin for build perf
                        },

timocov commented 6 years ago

We use LoaderOptionsPlugin to pass options. But I believe that we need to update to latest webpack/ts-loader first...

piecyk commented 6 years ago

Using webpack4 + thread-loader in mixed js/ts project, and it's working quite nicely... Basic have thread-loader, ts-loader, babel-loader, fork-ts-checker-webpack-plugin

Would recommend to have some custom code for configuring how many works use, interesting parts of the config...

module: {
  rules: [
    {
      ...commonRule,
      test: /\.tsx?$/,
      use: [
        tsWorkers > 0 && {
          loader: 'thread-loader',
          options: tsWorkerPool,
        },
        {
          loader: 'babel-loader',
          options: babelOptions({ hotMode, isDevelopmentMode }),
        },
        {
          loader: 'ts-loader',
          options: {
            happyPackMode: tsWorkers > 0,
            transpileOnly: tsTranspileOnly,
            compilerOptions: {
              sourceMap,
            },
          },
        },
      ].filter(Boolean),
    },
    {
      ...commonRule,
      test: /\.jsx?$/,
      use: [
        jsWorkers > 0 && {
          loader: 'thread-loader',
          options: jsWorkerPool,
        },
        {
          loader: 'babel-loader',
          options: babelOptions({ hotMode, isDevelopmentMode }),
        },
      ].filter(Boolean),
    },
    // ...
  ]
},
// fork-ts-checker-webpack-plugin in plugins 
plugins: [
  ...(tsTranspileOnly && forkTsWorkers > 0
    ? [
        new ForkTsCheckerWebpackPlugin({
          async: true,
          watch: [srcPath],
          workers: forkTsWorkers,
          checkSyntacticErrors: tsWorkers > 0,
        }),
      ]
    : []),
]

Workers pools

const commonModulesToWarmup = [
  'babel-loader',
  'babel-preset-env',
  'babel-preset-react',
  'babel-plugin-syntax-dynamic-import',
  'babel-plugin-syntax-object-rest-spread',
  'babel-plugin-transform-object-rest-spread',
]

const tsWorkerPool = {
  workers: tsWorkers,
  poolTimeout: watchMode || devServer ? Infinity : 2000,
}
if (tsWorkers > 0) {
  threadLoader.warmup(tsWorkerPool, [...commonModulesToWarmup, 'ts-loader'])
}
const jsWorkerPool = {
  workers: jsWorkers,
  poolTimeout: watchMode || devServer ? Infinity : 2000,
}
if (jsWorkers > 0) {
  threadLoader.warmup(jsWorkerPool, commonModulesToWarmup)
}

Running it with configuration of "tsWorkers":1,"jsWorkers":2,"forkTsWorkers":1 no blocking just building as mad 😄Plus thanks to hard-source-webpack-plugin running build from cache takes ~5s

felixfbecker commented 6 years ago

Can this be closed then? I don't think ts-loader should include this functionality if it can be achieved by composition of other loaders

piecyk commented 6 years ago

@felixfbecker Exactly 👍 There is no need for this logic in ts-loader

timocov commented 6 years ago

Actually there is one difference between using thread-loader and LanguageService in different thread - in case usage thread-loader the state between services is not shared and quite possible that some instances of ts-loader (and LanguageService) will make the same work independently.

piecyk commented 6 years ago

Yes that is true, just wondering in scenario that you use fork-ts-checker-webpack-plugin for type checking and thread-loader + ts-loader for transpileOnly will be it worth to run the LanguageService in other process 🤔

timocov commented 6 years ago

that you use fork-ts-checker-webpack-plugin for type checking

In some cases you just cannot use it - https://github.com/TypeStrong/ts-loader/issues/825#issuecomment-416956352

piecyk commented 6 years ago

Hmm what exactly this means ?

global const enums to inline some common constants

But yeah, then it make sens to run it in separate process... When running build without thread-loader & fork-ts-checker-webpack-plugin also gets blocked for few seconds

timocov commented 6 years ago

When running build without thread-loader & fork-ts-checker-webpack-plugin also gets blocked for few seconds

Could you please share a little bit about your project: how many files ts/js do you have, cloc value?

Hmm what exactly this means ?

Instead of declaring constants or importing const enums from module we just created local types, which contains global const enum declaration like this:

declare const enum KeyboardEventKeyCode {
    Backspace = 8,
    Tab = 9,
    Enter = 13,
    Shift = 16,
    Control = 17,
    Alt = 18,
    CapsLock = 20,
    Escape = 27,
    Space = 32,
    PageUp = 33,
    PageDown = 34,
    End = 35,
}

(this file is added to build the same as typing from @types folder)

And use it like this: if (e.keyCode === KeyboardEventKeyCode.End) {} where it needed in TypeScript code, and TypeScript will inline 35 instead of KeyboardEventKeyCode.End.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 5 years ago

Closing as stale. Please reopen if you'd like to work on this further.

TypeStrong / ts-loader

Try to move LanguageService to separate process #830