Document --incremental and composite project APIs

[DanielRosenwasser]

Follow-up from #29978, suggested by @MLoughry at https://github.com/microsoft/TypeScript/issues/29978#issuecomment-499674541

[ToddThomson]

Thank-you for the new incremental APIs in the upcoming Typescript 3.6!

I've read through the code, but cannot yet see how to plug in my own "compiler" to the solution builder so that I can make my own call to emit. My call to emit is needed to pass in transformers to the build. I can see how the "Watch" compiler is used from the builder, but that is not quite what I need.

@sheetalkamat If you could direct me down the right path to save me a bit of time I would be appreciative. Thanks in advance!

Edit:

Using getNextInvalidatedProject() looks promising and/or emitNextAffectedFile...

[sheetalkamat]

@ToddThomson The provision already exists and as you found out, If you need to pass custom transformers etc for emit you need to call getNextInvalidatedProject to get next project to build to get the project. It can return project to update time stamps on or just update --out file or project that needs build. The emit in the later two types of project does take custom transformers.

[ToddThomson]

@sheetalkamat Thank-you. The use of getNextInvalidatedProject seems quite clear.

It was my assumption that overriding emitNextAffectedFile during host.createProgram would allow custom transformers to be introduced to the solution builder pipeline when using build(). This does not work, but I feel that it should. I'd be interested to know why.

[sheetalkamat]

That's because in build is for all the projects and its not clear if the transformers are suppose to be for one project or for all..

[ToddThomson]

@sheetalkamat That makes sense. Perhaps transformers should be part of the project configuration file then.

Thanks for your help and have a great day!

[hipstersmoothie]

@sheetalkamat I'm having trouble getting my custom compiler to actually use the buildInfo files to shorten the builds

      const host = ts.createIncrementalCompilerHost(allOptions, ts.sys);
      const program = ts.createIncrementalProgram({
        host,
        options: allOptions,
        projectReferences,
        rootNames: fileNames
      });

      const { diagnostics: emitDiagnostics } = program.emit();

my options look like

{ target: 1,
  module: 1,
  strict: true,
  jsx: 1,
  allowSyntheticDefaultImports: true,
  sourceMap: true,
  declaration: true,
  lib:
   [ 'lib.es2015.d.ts',
     'lib.es2017.d.ts',
     'lib.dom.d.ts',
     'lib.es2019.d.ts' ],
  esModuleInterop: true,
  experimentalDecorators: true,
  resolveJsonModule: true,
  downlevelIteration: true,
  outDir: 'dist',
  rootDir: '/Users/alisowski/Documents/cgds/components/Card/src',
  composite: true,
  configFilePath:
   '/Users/alisowski/Documents/cgds/components/Card/tsconfig.json',
  incremental: true,
  emitDeclarationOnly: true }

any guidance? when I run tsc with the same setup it uses the buildinfo files

[ulrichb]

@sheetalkamat I would also need some hint.

I tried this:

const host = ts.createIncrementalCompilerHost(compilerOptions);
const options: ts.CompilerOptions = {
    ...compilerOptions,
    incremental: true,
    tsBuildInfoFile: "tsBuildInfo.json"
};

const program = ts.createIncrementalProgram({ rootNames: filesToCompile, options, host });

let nextAffected;
while ((nextAffected = program.emitNextAffectedFile()) !== undefined) {
    const result = nextAffected.result;
    console.log(result.emitSkipped); // Always false :(
    console.log(result.diagnostics);
}

return [
    ...program.getOptionsDiagnostics(), ...program.getGlobalDiagnostics(), ...program.getSyntacticDiagnostics(),
    ...program.getDeclarationDiagnostics(), ...program.getSemanticDiagnostics(),
];

... and I also get a written tsBuildInfo.json, but it doesn't seem to be used on re-compilations: The "cold start" time is the same as without incremental compilation and result.emitSkipped is always false.

[sheetalkamat]

@sheetalkamat I'm having trouble getting my custom compiler to actually use the buildInfo files to shorten the builds const host = ts.createIncrementalCompilerHost(allOptions, ts.sys); const program = ts.createIncrementalProgram({ host, options: allOptions, projectReferences, rootNames: fileNames });

  const { diagnostics: emitDiagnostics } = program.emit();

my options look like { target: 1, module: 1, strict: true, jsx: 1, allowSyntheticDefaultImports: true, sourceMap: true, declaration: true, lib: [ 'lib.es2015.d.ts', 'lib.es2017.d.ts', 'lib.dom.d.ts', 'lib.es2019.d.ts' ], esModuleInterop: true, experimentalDecorators: true, resolveJsonModule: true, downlevelIteration: true, outDir: 'dist', rootDir: '/Users/alisowski/Documents/cgds/components/Card/src', composite: true, configFilePath: '/Users/alisowski/Documents/cgds/components/Card/tsconfig.json', incremental: true, emitDeclarationOnly: true } any guidance? when I run tsc with the same setup it uses the buildinfo files

I am not sure what could be going wrong.. Try debugging and see if tsbuildInfo that is being read. If it is check if version written there is same as ts.version and go from there?

[sheetalkamat]

const host = ts.createIncrementalCompilerHost(compilerOptions); const options: ts.CompilerOptions = { ...compilerOptions, incremental: true, tsBuildInfoFile: "tsBuildInfo.json" };

You are creating host with wrong options so those options are getting used to determine the tsbuild info path or something else in the compiler.

while ((nextAffected = program.emitNextAffectedFile()) !== undefined) { const result = nextAffected.result; console.log(result.emitSkipped); // Always false :( console.log(result.diagnostics); }

result.emitSkipped in most cases is going to be always false since it only emits the required files and emit for those almost always is expected to succeed

[ulrichb]

result.emitSkipped in most cases is going to be always false

Okay, so how can I determine which files have been really skipped in the incremental (follow-up) build?

I'm only asking for debugging purposes. What I really want is better cold-start performance 😃. And atm. I don't see a difference; so I assume that the incremental build info isn't used at all (although written).

Another observation: In ProcMon, I saw "WriteFile" events for all .js and .js.map files (without any changes to the input .ts files). Further, I checked the tsBuildInfo.json content (which also always gets written): It's binary same after a recompilation. So I'm doing something wrong here :/

BTW: Using TS 3.6.1 RC

[sheetalkamat]

Okay, so how can I determine which files have been really skipped in the incremental (follow-up) build?

You will need to track of what files are printed when doing clean build and from there deduce which are not built, because emit on builder does not provide that information. It just emits the files that need to be.. You can see which files are emitted by passing --listEmittedFiles option

I'm only asking for debugging purposes. What I really want is better cold-start performance 😃. And atm. I don't see a difference; so I assume that the incremental build info isn't used at all (although written).

You would need to see if buildInfo is being read or not? and if it is then check the version if it is matched. (add host.readFile to override that lets you check if buildinfo is read or not. You would want to ) (i noticed that in earlier response o had commented on your options being not set correctly but its not reflected correctly and went into block scope. I have updated it so make sure you look into that as well.

[hipstersmoothie]

I have made an example repo showcasing the issue. If I am reading correctly it seems like we have to manage the "short circuit" ourselves by watching output? I would assume that this is handled createIncrementalCompilerHost or createIncrementalProgram

https://github.com/hipstersmoothie/typescript-incremental-node-compiler-example

[hipstersmoothie]

I added the following to my repo and got some interesting results

const host = ts.createIncrementalCompilerHost(allOptions, {
  ...ts.sys,
  readFile: (...args) => {
    return ts.sys.readFile(...args);
  }
});

It seems to be reading the tsconfig.tsbuildinfo from the wrong place

READ FILE tsconfig.tsbuildinfo
READ FILE /Users/alisowski/Documents/incremental-example/src/index.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es5.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.core.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.collection.d.ts

[hipstersmoothie]

Even if i pass an absolute path to my buildinfo file it still reads the other files

const allOptions = {
  ...options,
  outDir: 'dist',
  incremental: true,
  emitDeclarationOnly: true,
  listEmittedFiles: true,
  tsBuildInfoFile: path.join(__dirname, 'tsconfig.tsbuildinfo')
};

READ FILE /Users/alisowski/Documents/incremental-example/tsconfig.tsbuildinfo
READ FILE /Users/alisowski/Documents/incremental-example/src/index.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es5.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.core.d.ts

[hipstersmoothie]

Also to help me debug: where exactly does this short circuit happen? I'm having trouble finding where in the code it actually cancels the build based on the build info

[hipstersmoothie]

I have updated the example with the above findings. Now my example will not rebuild the files (yay!) but my build times are still about the same. Comparing with tsc the difference is stark

To check that the builds are fast run

yarn build - (get initial emit) ~2.5s
yarn build -  (second emit should be much shorter) ~2.5s

Compare this with tsc:

yarn tsc -b tsconfig.json --incremental  - (first time takes ~2s)
yarn tsc -b tsconfig.json --incremental  - (second time takes 0.31s)

[ulrichb]

I also had no luck. I've added a console.log in host's .readFile() .getSourceFile() and .writeFile() and the result:

• 1 x readFile to tsBuildInfo.json
• getSourceFile() for all the (indirectly) referenced (lib) .d.ts files and all source files
• writeFile() for all .js and .js.map files
• 1 x writeFile() to tsBuildInfo.json

[ulrichb]

@sheetalkamat Do I need to manually call readBuilderProgram() and pass the oldProgram somehow into createIncrementalProgram()?

[hipstersmoothie]

@sheetalkamat any pointers? I feel like i'm almost there but now my wheels are spinning

[hipstersmoothie]

I just upgraded my example repo to 3.6.2.

My compile times now look like:

To check that the builds are fast run

yarn build - (get initial emit) ~1.65s
yarn build -  (second emit should be much shorter) ~1.65s

Compare this with tsc:

yarn build:tsc - (first time takes ~1.55s)
yarn build:tsc - (second time takes 0.27s)

Is there a working example anywhere (other than tsc) where i can see incremental builds via the node API?

[sheetalkamat]

@hipstersmoothie I am looking into your repro and seeing if there is bug. @ulrichb you don't need to call readBuilderProgram explicitly. createIncrementalProgram does that for you.

[sheetalkamat]

@hipstersmoothie it seems like you are patching tsbuildInfoFile at https://github.com/hipstersmoothie/typescript-incremental-node-compiler-example/blob/master/compiler.js#L17 but the path isn't set correctly (When we read config file, compiler options are updated with absolute paths with / instead or \. Because the tsbuildInfoFile options affects what will be emitted the files, all the files are emitted again and hence you are not seeing the gains.

[ulrichb]

@sheetalkamat I had also backslashes in my tsbuildInfoFile path. Replaced them with slashes => still no effect. All .js files get written (although all input hashes are stable) => no speed-up.

Could you maybe provide a minimal sample with a simple (single-project) incremental build using createIncrementalProgram()?

[hipstersmoothie]

@sheetalkamat Updated my repo (removed line 17) and still see no incremental compilation

[sheetalkamat]

@hipstersmoothie there are two issues with your usage. You are calling emit before getting diagnostics so semantic diagnostics are cached after emitting tsbuildInfo file so tsbuildInfo does not contain the semantic diagnostics. If you fix the order there is another issue that you are calling getPreEmitDiagnostics which also tries to get declarationDiagnostics since you have that enabled and those are calculated by doing emit in memory so you aren't saving emit time either..

[hipstersmoothie]

Okay that makes sense. What is the best way for me to collect all the diagnostics in a --incremental friendly way?

[sheetalkamat]

const diagnostics = [
    ...program.getConfigFileParsingDiagnostics(),
    ...program.getSyntacticDiagnostics(),
    ...program.getOptionsDiagnostics(),
    ...program.getOptionsDiagnostics(),
    ...program.getSemanticDiagnostics()
];
const result = program.emit();
const allDiagnostics = diagnostics.concat(result.diagnostics);

Note that you would need build with #33170 for 3.6 to be able to actually do incremental semantic diagnostics and its not yet in build from that branch. You can in the mean while test with typescript@next

[ulrichb]

2x program.getOptionsDiagnostics() ?

[hipstersmoothie]

Okay now I'm actually seeing a difference in compile times!

yarn build - (get initial emit) ~2.3s
yarn build -  (second emit should be much shorter) ~1.12s

Compare this with tsc:

yarn build:tsc - (first time takes ~1.80s)
yarn build:tsc - (second time takes 0.35s)

However tsc is still a lot faster

[sheetalkamat]

you aren't comparing correct times.. build:tsc uses --b option which is different api from incremental api you are using. The api you are using is for tsc -p tsconfig.json --incremental.

[hipstersmoothie]

Oh interesting. Is there a way to use the -b API?

[ulrichb]

@sheetalkamat Many thanks for the sample in the Wiki!

I have got it now too. My problem was that I created the ts.CompilerOptions "on the fly" (instead of using getParsedCommandLineOfConfigFile()) with a literal, and the problem was that configFilePath was missing (automatically added by getParsedCommandLineOfConfigFile()).

I have now { ...otherOptions, incremental: true, configFilePath: "tsconfig.json" } and now everything works as expected (~ 50% compilation time, no more unnecessary file-writes).

Note that a) configFilePath points to a "tsconfig.json" which doesn't even have to exist, but TS uses it to generate the file name tsconfig.tsbuildinfo, and b) tsBuildInfoFile: "somefile.json" alone (without configFilePath) does not work (although TS writes the tsBuildInfoFile, it doesn't use it on following compiles).

So @sheetalkamat, is this a bug (using 3.7.0-dev.20190903)?

[sheetalkamat]

So @sheetalkamat, is this a bug (using 3.7.0-dev.20190903)?

@ulrichb Not sure if its configuration issue. Please provide complete steps to repro, probably a repo where this repros and we can take a look.

[hipstersmoothie]

@sheetalkamat Is there an API to replicate -b? I'd really like to squeeze out that last little bit of time

[hipstersmoothie]

I see the new docs and it's working! (I think)

One last question is: how to feed custom transformers to the solution builder?

[sheetalkamat]

I see the new docs and it's working! (I think) One last question is: how to feed custom transformers to the solution builder?

You can pass that to done or emit of project as shown in alternative method building solution: https://github.com/microsoft/TypeScript-wiki/pull/225/files#diff-709351cd55688fbcb7ec0fc9973ee746R407

[hipstersmoothie]

Perfect! Thanks for all the help

[hipstersmoothie]

I can't seem to get only emitting .d.ts files to work though. It goes into an infinite loop

const host = ts.createSolutionBuilderHost(
  undefined,
  undefined,
  reportDiagnostic,
  reportSolutionBuilderStatus,
  reportErrorSummary
);

const solution = ts.createSolutionBuilder(host, ['./tsconfig.json'], {
  verbose: true,
  emitDeclarationOnly: true
});

while (true) {
  const project = solution.getNextInvalidatedProject();

  if (!project) {
    break;
  }

  project.emit(undefined, undefined, undefined, true);
}

[ulrichb]

@sheetalkamat While trying to create a repro sample, I mentioned that it also works using just tsBuildInfoFile (without a dummy configFilePath), but you must use an absolute dir and slashes instead of backslashes for both outDir and tsBuildInfoFile:

const compilerOptions = {
    // ...
    outDir: outDirAbsolute.replace(/\\/g, "/"),
    incremental: true,
    tsBuildInfoFile: path.resolve(outDirAbsolute, "tsconfig.tsbuildinfo").replace(/\\/g, "/"),
};

So, I'm happy now. Many thanks again!

BTW: Regarding cross platform compat, the "path normalization requirement" is a bug waiting to happen when starting with Linux/macOS and later running on Windows.

[hipstersmoothie]

@sheetalkamat seems like on this line https://github.com/microsoft/TypeScript/blob/master/src/compiler/tsbuild.ts#L433 it filters out any extra CompilerOptions (in my case emitDeclarationOnly). If i just set baseCompilerOptions= option it all works as expected

EDIT: I can also do this, but it feels very, very dirty.

ts.commonOptionsWithBuild.push({ name: 'emitDeclarationOnly' })
const solution = ts.createSolutionBuilder(host, ['./tsconfig.json'], {
  verbose: true,
  incremental: true,
  listEmittedFiles: true,
  emitDeclarationOnly: true
});

[sheetalkamat]

@sheetalkamat seems like on this line https://github.com/microsoft/TypeScript/blob/master/src/compiler/tsbuild.ts#L433 it filters out any extra CompilerOptions (in my case emitDeclarationOnly). If i just set baseCompilerOptions= option it all works as expected

That is intended behavior, BuildOptions are different from CompilerOptions and only subset of those are allowed.

[hipstersmoothie]

So there is no way to pass compiler options and no future support?

[hipstersmoothie]

Is it intended that customTransformers do not run when emitDeclarationOnly is true?

[hipstersmoothie]

It seems i could use afterDeclarations but that doesn't get and AST with all the code, just the stuff that ends up in .d.ts

EDIT: Fixed that. The .d.ts ast has a reference to the source

Now i'm just trying to get my custom transformer to abort the emit

[ahnpnl]

I can't seem to get only emitting .d.ts files to work though. It goes into an infinite loop

const host = ts.createSolutionBuilderHost(
  undefined,
  undefined,
  reportDiagnostic,
  reportSolutionBuilderStatus,
  reportErrorSummary
);

const solution = ts.createSolutionBuilder(host, ['./tsconfig.json'], {
  verbose: true,
  emitDeclarationOnly: true
});

while (true) {
  const project = solution.getNextInvalidatedProject();

  if (!project) {
    break;
  }

  project.emit(undefined, undefined, undefined, true);
}

I have the same issue. Have you managed to make it work ?

[inad9300]

I've had some success following all your indications here, and the new documentation (thank you!) However, I find that incremental compilation does not work when using the outFile option. The *.tsbuildinfo file becomes much more terse, and compilation times are comparable to not having the incremental flag turned on.

Here is an example of a main.tsbuildinfo file generated by TypeScript when trying to compile a single-file program (main.ts) using incremental compilation, AMD as module, and the outFile option:

{
  "bundle": {
    "commonSourceDirectory": "../../src",
    "sourceFiles": [
      "../../src/main.ts"
    ],
    "js": {
      "sections": [
        {
          "pos": 0,
          "end": 13,
          "kind": "prologue",
          "data": "use strict"
        },
        {
          "pos": 14,
          "end": 44,
          "kind": "text"
        }
      ],
      "sources": {
        "prologues": [
          {
            "file": 0,
            "text": "",
            "directives": [
              {
                "pos": -1,
                "end": -1,
                "expression": {
                  "pos": -1,
                  "end": -1,
                  "text": "use strict"
                }
              }
            ]
          }
        ]
      }
    }
  },
  "version": "3.8.2"
}

When removing the outFile option, the filename ends up being tsconfig.tsbuildinfo instead, and is 1550 lines long.

[RomainMuller]

How does one implement --build --watch behavior and use custom transformers?

[hipstersmoothie]

Here's how I ended up doing it: https://github.com/intuit/design-systems-cli/blob/master/plugins/build/src/typescript.ts#L173

[RomainMuller]

@hipstersmoothie that's pretty similar to what I was doing out of spite... Figured there should be a better way; but it juts looks like the necessary invalidation logic is presently burried.

[mortyccp]

Hi. I am building a jest transformer that use solution builder. And I need some help on how can I get the emitted file for the jest input sourcePath. 🙏

import type { Transformer } from "@jest/transform";

import * as ts from "typescript";

const transformer: Transformer = {
  process: (sourceText, sourcePath, config, options) => {
    console.log({ sourcePath, rootDir: config.rootDir });

    const host = ts.createSolutionBuilderHost({
      ...ts.sys,
      readFile(path: string, encoding?: string) {
        console.log("readFile", { path, encoding });
        return ts.sys.readFile(path, encoding);
      },
    });
    const solutionBuilder = ts.createSolutionBuilder(
      host,
      [config.rootDir],
      {}
    );
    while (true) {
      const next = solutionBuilder.getNextInvalidatedProject();
      if (next === undefined) {
        break;
      }
      switch (next.kind) {
        case ts.InvalidatedProjectKind.Build:
          next.done();
          break;
        case ts.InvalidatedProjectKind.UpdateBundle:
          next.done();
          break;
        case ts.InvalidatedProjectKind.UpdateOutputFileStamps:
          next.done();
          break;
      }
    }

    // TODO: Get the emitted files for the sourcePath

    return {
      code: "",
      map: "",
    };
  },
};

export const process = transformer.process;