Optimal TS monorepos: how to handle project references when projects opt out of composite within dependency chain

JamesHenry commented 3 weeks ago

🔎 Search Terms

project references no output

🕗 Version & Regression Information

N/A

⏯ Playground Link

N/A

💻 Code

// Your code here

🙁 Actual behavior

Please see full description, repro and video walkthrough

🙂 Expected behavior

Please see full description, repro and video walkthrough

Additional information about the issue

I am reporting this as an issue because @jakebailey advised me that that would make sense for any time a Debug Error is reached in TS, but also to get formal guidance from the TS Team on finalizing our new TypeScript plugin for Nx.

As discussed in our sync between our teams a couple of months ago, we are building a new mechanism to sync Nx project graph (derived from package.json files and import and export statements) automatically to configuration files on disk. This is a generic mechanism, but in the context of TS, this applies to syncing project references in tsconfig.json files.

This should allow us to keep Nx out of the way of tsc where appropriate (tsc will be directly executed by Nx behind the scenes when using nx typecheck my-ts-proj or nx build my-ts-proj for example) and still allow users to run raw tsc if they wish because all the relevant config will live on disk and be up to date thanks to Nx.

The syncing mechanism is working really well already for repo's which can fully stick to composite: true throughout the codebase, but we have already run into one case on the https://github.com/typescript-eslint/typescript-eslint (where @jakebailey and @sheetalkamat kindly help out) where a couple of projects needed to opt out of composite: true and not emit declaration files at all.

As explained in detail in attached loom covering the reproduction repo, we would like some advice on how we should attempt to handle cases like this in our syncing logic.

The super brief summary of the issue causing it to hit the Debug Failure is:

Projects: A -> B -> C
B is opting out of composite
The root solution file can happily reference all 3 projects as hoped, despite the composite opt out of B
We want to be able to typecheck A, B and C
tsc -b --emitDeclarationOnly is used for typechecking (intentionally kept distinct from building with tsc, producing JS artifacts)
When typechecking the whole repo, or typechecking just A, we hit the Debug failure because of B not having any outputs before proceeding onto A

🎥 Context and issue reproduction: https://www.loom.com/share/84a7098b3d1e4a819bd598ad7fc3ae7f

🔗 Reproduction repo: https://github.com/JamesHenry/tsc-composite-false-repro

jakebailey commented 3 weeks ago

Can you post your crashing stack trace and instructions that go with that repo?

(A video is not the most convenient way to check a repro)

JamesHenry commented 3 weeks ago

For sure, sorry, I should have included explicit steps too, although the video establishes/reiterates the "why" behind the approach and failing command which is important to avoid a simple "don't pass --emitDeclarationOnly" resolution to this.

The Debug failure kind of makes sense based on the setup, but hopefully it's clear the issue isn't just about the debug failure being hit but also the need for a recommendation on how to handle monorepos with these concerns (any one or more projects opting out of declaration generation but needing to be typechecked alongside all the other projects within the monorepo).

Commands:

npm install
npx tsc -b --verbose --emitDeclarationOnly

Stack trace:

❯ npx tsc -b --verbose --emitDeclarationOnly
[10:01:05 PM] Projects in this build: 
    * packages/c/tsconfig.build.json
    * packages/c/tsconfig.spec.json
    * packages/c/tsconfig.json
    * packages/b/tsconfig.build.json
    * packages/b/tsconfig.spec.json
    * packages/b/tsconfig.json
    * packages/a/tsconfig.build.json
    * packages/a/tsconfig.spec.json
    * packages/a/tsconfig.json
    * tsconfig.json

[10:01:05 PM] Project 'packages/c/tsconfig.build.json' is out of date because output file 'dist/packages/c/tsconfig.build.tsbuildinfo' does not exist

[10:01:05 PM] Building project '/Users/james/Repos/tmp/tsc-composite-false-repro/packages/c/tsconfig.build.json'...

[10:01:05 PM] Project 'packages/c/tsconfig.spec.json' is out of date because output file 'dist/out-tsc/packages/a/tsconfig.spec.tsbuildinfo' does not exist

[10:01:05 PM] Building project '/Users/james/Repos/tmp/tsc-composite-false-repro/packages/c/tsconfig.spec.json'...

[10:01:05 PM] Project 'packages/b/tsconfig.build.json' is up to date with .d.ts files from its dependencies

/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:1280
    throw e;
    ^

Error: Debug Failure. project /Users/james/Repos/tmp/tsc-composite-false-repro/packages/b/tsconfig.build.json expected to have at least one output
    at getFirstProjectOutput (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:113045:16)
    at updateOutputTimestamps (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:127725:27)
    at Object.done (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:126896:9)
    at buildWorker (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:127787:24)
    at build (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:127772:18)
    at Object.build (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:128053:79)
    at performBuild (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:128964:69)
    at executeCommandLine (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:128868:14)
    at Object.<anonymous> (/Users/james/Repos/tmp/tsc-composite-false-repro/node_modules/.pnpm/typescript@5.5.4/node_modules/typescript/lib/tsc.js:129809:1)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)

Node.js v20.12.1

sheetalkamat commented 1 week ago

Sorry for delay but I was on vacation and have been thinking about this today

I will definitely take a look at debug failure to see what’s going on there

but i have also been thinking about this problem of disabling composite to disable declaration emit..

Right now composite means three things

emit declarations because they are needed to be used by projects that reference them
list all ts files and ensure that they fall in root dir, so we know the complete list of input and output to check for timestamps and file contents for up to date ness
Incremental so we can build these project incrementally but also store the extra information about the latest data file stamp to look and that helps us not having to check contents of all data files and timestamps .

All these things make referencing project to not build frequently and do fast up to dateness check.

So if you passed emit declarations on command line, we can’t guarantee 2 and 3 .

We do have a way around type checking using source files instead of dts and that works well in editor most of the time with some caveats where you might not get correct type or your program might get bigger etc .. it also have implications on outputs esp dts with types not narrowed down as expected ..

So given this is bigger design issue , I am still listing points I have thought without giving any recommendation or suggestion on how we can fix this .. I will think about this some more before I come up with proposal for sure since my thoughts are still raw.

microsoft / TypeScript