"Unable to build typescript at rev 3d8cf62846c8bda828675239ddd9d4bd33e53d51"

[DavidArchibald]

Hey! Thanks for the great tool! Saved me plenty of time already finding which PRs break my code.

Anyways, I was running a bisect and I got this error Unable to build typescript at rev 3d8cf62846c8bda828675239ddd9d4bd33e53d51; please file a bug! Just in case something funky was happening with how I'd set up my bisect, I stopped my bisect and then ran every-ts switch 3d8cf62846c8bda828675239ddd9d4bd33e53d51 and sure enough I got the error again.

Here's the full logs:

Updating files: 100% (49100/49100), done.
Previous HEAD position was d23b7e7c52 export TypingsInstaller from tsserverlibrary (#53394)
HEAD is now at 3d8cf62846 Merge pull request #16463 from Microsoft/jsdoc-@template-in-scope-as-type-parameter
Building TypeScript...
Error: Command failed with exit code 1: /home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/fnm/fnm exec --using=8 -- node_modules/.bin/jake local
jake aborted.
Error: ENOENT: no such file or directory, unlink 'built/local/typingsInstaller.js'
    at Object.fs.unlinkSync (fs.js:1061:18)
    at Exec.<anonymous> (/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/TypeScript/Jakefile.js:374:16)
(See full trace by running task with --trace)
node lib/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --preserveConstEnums --out scripts/processDiagnosticMessages.js -sourcemap --target es5 --lib es5 --noUnusedLocals --noUnusedParameters scripts/processDiagnosticMessages.ts

node scripts/processDiagnosticMessages.js src/compiler/diagnosticMessages.json
node lib/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --preserveConstEnums --out built/local/tsc.js -sourcemap --target es5 --lib es5 --noUnusedLocals --noUnusedParameters src/compiler/core.ts src/compiler/performance.ts src/compiler/sys.ts src/compiler/types.ts src/compiler/scanner.ts src/compiler/parser.ts src/compiler/utilities.ts src/compiler/binder.ts src/compiler/checker.ts src/compiler/factory.ts src/compiler/visitor.ts src/compiler/transformers/ts.ts src/compiler/transformers/jsx.ts src/compiler/transformers/esnext.ts src/compiler/transformers/es2017.ts src/compiler/transformers/es2016.ts src/compiler/transformers/es2015.ts src/compiler/transformers/es5.ts src/compiler/transformers/generators.ts src/compiler/transformers/destructuring.ts src/compiler/transformers/module/module.ts src/compiler/transformers/module/system.ts src/compiler/transformers/module/es2015.ts src/compiler/transformer.ts src/compiler/comments.ts src/compiler/sourcemap.ts src/compiler/declarationEmitter.ts src/compiler/emitter.ts src/compiler/program.ts src/compiler/commandLineParser.ts src/compiler/tsc.ts src/compiler/diagnosticInformationMap.generated.ts

node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --declaration --preserveConstEnums --out built/local/typescriptServices.js -sourcemap --stripInternal --target es5 --lib es5 --noUnusedLocals --noUnusedParameters src/services/../compiler/core.ts src/services/../compiler/performance.ts src/services/../compiler/sys.ts src/services/../compiler/types.ts src/services/../compiler/scanner.ts src/services/../compiler/parser.ts src/services/../compiler/utilities.ts src/services/../compiler/binder.ts src/services/../compiler/checker.ts src/services/../compiler/factory.ts src/services/../compiler/visitor.ts src/services/../compiler/transformers/ts.ts src/services/../compiler/transformers/jsx.ts src/services/../compiler/transformers/esnext.ts src/services/../compiler/transformers/es2017.ts src/services/../compiler/transformers/es2016.ts src/services/../compiler/transformers/es2015.ts src/services/../compiler/transformers/es5.ts src/services/../compiler/transformers/generators.ts src/services/../compiler/transformers/destructuring.ts src/services/../compiler/transformers/module/module.ts src/services/../compiler/transformers/module/system.ts src/services/../compiler/transformers/module/es2015.ts src/services/../compiler/transformer.ts src/services/../compiler/comments.ts src/services/../compiler/sourcemap.ts src/services/../compiler/declarationEmitter.ts src/services/../compiler/emitter.ts src/services/../compiler/program.ts src/services/../compiler/commandLineParser.ts src/services/../compiler/diagnosticInformationMap.generated.ts src/services/types.ts src/services/utilities.ts src/services/breakpoints.ts src/services/classifier.ts src/services/completions.ts src/services/documentHighlights.ts src/services/documentRegistry.ts src/services/findAllReferences.ts src/services/importTracker.ts src/services/goToDefinition.ts src/services/jsDoc.ts src/services/jsTyping.ts src/services/navigateTo.ts src/services/navigationBar.ts src/services/outliningElementsCollector.ts src/services/pathCompletions.ts src/services/patternMatcher.ts src/services/preProcess.ts src/services/rename.ts src/services/services.ts src/services/transform.ts src/services/transpile.ts src/services/shims.ts src/services/signatureHelp.ts src/services/symbolDisplay.ts src/services/textChanges.ts src/services/refactorProvider.ts src/services/codeFixProvider.ts src/services/formatting/formatting.ts src/services/formatting/formattingContext.ts src/services/formatting/formattingRequestKind.ts src/services/formatting/formattingScanner.ts src/services/formatting/references.ts src/services/formatting/rule.ts src/services/formatting/ruleAction.ts src/services/formatting/ruleDescriptor.ts src/services/formatting/ruleFlag.ts src/services/formatting/ruleOperation.ts src/services/formatting/ruleOperationContext.ts src/services/formatting/rules.ts src/services/formatting/rulesMap.ts src/services/formatting/rulesProvider.ts src/services/formatting/smartIndenter.ts src/services/formatting/tokenRange.ts src/services/codefixes/disableJsDiagnostics.ts src/services/codefixes/fixAddMissingMember.ts src/services/codefixes/fixClassDoesntImplementInheritedAbstractMember.ts src/services/codefixes/fixClassIncorrectlyImplementsInterface.ts src/services/codefixes/fixClassSuperMustPrecedeThisAccess.ts src/services/codefixes/fixConstructorForDerivedNeedSuperCall.ts src/services/codefixes/fixExtendsInterfaceBecomesImplements.ts src/services/codefixes/fixForgottenThisPropertyAccess.ts src/services/codefixes/fixSpelling.ts src/services/codefixes/fixes.ts src/services/codefixes/helpers.ts src/services/codefixes/importFixes.ts src/services/codefixes/unusedIdentifierFixes.ts src/services/refactors/convertFunctionToEs6Class.ts src/services/refactors/refactors.ts

node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types node --pretty --preserveConstEnums --outDir built/local/ --module commonjs -sourcemap --target es5 --lib es6 --noUnusedLocals --noUnusedParameters src/server/cancellationToken/cancellationToken.ts

node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types node --pretty --preserveConstEnums --outDir built/local/ --out built/local/typingsInstaller.js -sourcemap --target es5 --lib es6 --noUnusedLocals --noUnusedParameters src/server/typingsInstaller/../types.ts src/server/typingsInstaller/../shared.ts src/server/typingsInstaller/typingsInstaller.ts src/server/typingsInstaller/nodeTypingsInstaller.ts

125             process.on("message", (req: DiscoverTypings | CloseProject) => {
                           ~~~~~~~~~

src/server/typingsInstaller/nodeTypingsInstaller.ts(125,24): error TS2345: Argument of type '"message"' is not assignable to parameter of type 'Signals'.

    at makeError (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/node_modules/execa/lib/error.js:60:11)
    at handlePromise (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/node_modules/execa/index.js:124:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildFuncs (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/repo.js:106:9)
    at async tryBuildFns (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/repo.js:125:13)
    at async ensureBuiltWorker (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/repo.js:159:9)
    at async ensureBuilt (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/repo.js:173:26)
    at async Switch.executeOrThrow (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/git.js:146:9)
    at async Switch.execute (file:///home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/dist/common.js:50:20)
    at async Switch.validateAndExecute (/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/node_modules/clipanion/lib/advanced/Command.js:73:26) {
  shortMessage: 'Command failed with exit code 1: /home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/fnm/fnm exec --using=8 -- node_modules/.bin/jake local',
  command: '/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/fnm/fnm exec --using=8 -- node_modules/.bin/jake local',
  escapedCommand: '"/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/fnm/fnm" exec "--using=8" -- "node_modules/.bin/jake" local',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: 'node lib/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --preserveConstEnums --out scripts/processDiagnosticMessages.js -sourcemap --target es5 --lib es5 --noUnusedLocals --noUnusedParameters scripts/processDiagnosticMessages.ts\n' +
    '\n' +
    'node scripts/processDiagnosticMessages.js src/compiler/diagnosticMessages.json\n' +
    'node lib/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --preserveConstEnums --out built/local/tsc.js -sourcemap --target es5 --lib es5 --noUnusedLocals --noUnusedParameters src/compiler/core.ts src/compiler/performance.ts src/compiler/sys.ts src/compiler/types.ts src/compiler/scanner.ts src/compiler/parser.ts src/compiler/utilities.ts src/compiler/binder.ts src/compiler/checker.ts src/compiler/factory.ts src/compiler/visitor.ts src/compiler/transformers/ts.ts src/compiler/transformers/jsx.ts src/compiler/transformers/esnext.ts src/compiler/transformers/es2017.ts src/compiler/transformers/es2016.ts src/compiler/transformers/es2015.ts src/compiler/transformers/es5.ts src/compiler/transformers/generators.ts src/compiler/transformers/destructuring.ts src/compiler/transformers/module/module.ts src/compiler/transformers/module/system.ts src/compiler/transformers/module/es2015.ts src/compiler/transformer.ts src/compiler/comments.ts src/compiler/sourcemap.ts src/compiler/declarationEmitter.ts src/compiler/emitter.ts src/compiler/program.ts src/compiler/commandLineParser.ts src/compiler/tsc.ts src/compiler/diagnosticInformationMap.generated.ts\n' +
    '\n' +
    'node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types  --pretty --declaration --preserveConstEnums --out built/local/typescriptServices.js -sourcemap --stripInternal --target es5 --lib es5 --noUnusedLocals --noUnusedParameters src/services/../compiler/core.ts src/services/../compiler/performance.ts src/services/../compiler/sys.ts src/services/../compiler/types.ts src/services/../compiler/scanner.ts src/services/../compiler/parser.ts src/services/../compiler/utilities.ts src/services/../compiler/binder.ts src/services/../compiler/checker.ts src/services/../compiler/factory.ts src/services/../compiler/visitor.ts src/services/../compiler/transformers/ts.ts src/services/../compiler/transformers/jsx.ts src/services/../compiler/transformers/esnext.ts src/services/../compiler/transformers/es2017.ts src/services/../compiler/transformers/es2016.ts src/services/../compiler/transformers/es2015.ts src/services/../compiler/transformers/es5.ts src/services/../compiler/transformers/generators.ts src/services/../compiler/transformers/destructuring.ts src/services/../compiler/transformers/module/module.ts src/services/../compiler/transformers/module/system.ts src/services/../compiler/transformers/module/es2015.ts src/services/../compiler/transformer.ts src/services/../compiler/comments.ts src/services/../compiler/sourcemap.ts src/services/../compiler/declarationEmitter.ts src/services/../compiler/emitter.ts src/services/../compiler/program.ts src/services/../compiler/commandLineParser.ts src/services/../compiler/diagnosticInformationMap.generated.ts src/services/types.ts src/services/utilities.ts src/services/breakpoints.ts src/services/classifier.ts src/services/completions.ts src/services/documentHighlights.ts src/services/documentRegistry.ts src/services/findAllReferences.ts src/services/importTracker.ts src/services/goToDefinition.ts src/services/jsDoc.ts src/services/jsTyping.ts src/services/navigateTo.ts src/services/navigationBar.ts src/services/outliningElementsCollector.ts src/services/pathCompletions.ts src/services/patternMatcher.ts src/services/preProcess.ts src/services/rename.ts src/services/services.ts src/services/transform.ts src/services/transpile.ts src/services/shims.ts src/services/signatureHelp.ts src/services/symbolDisplay.ts src/services/textChanges.ts src/services/refactorProvider.ts src/services/codeFixProvider.ts src/services/formatting/formatting.ts src/services/formatting/formattingContext.ts src/services/formatting/formattingRequestKind.ts src/services/formatting/formattingScanner.ts src/services/formatting/references.ts src/services/formatting/rule.ts src/services/formatting/ruleAction.ts src/services/formatting/ruleDescriptor.ts src/services/formatting/ruleFlag.ts src/services/formatting/ruleOperation.ts src/services/formatting/ruleOperationContext.ts src/services/formatting/rules.ts src/services/formatting/rulesMap.ts src/services/formatting/rulesProvider.ts src/services/formatting/smartIndenter.ts src/services/formatting/tokenRange.ts src/services/codefixes/disableJsDiagnostics.ts src/services/codefixes/fixAddMissingMember.ts src/services/codefixes/fixClassDoesntImplementInheritedAbstractMember.ts src/services/codefixes/fixClassIncorrectlyImplementsInterface.ts src/services/codefixes/fixClassSuperMustPrecedeThisAccess.ts src/services/codefixes/fixConstructorForDerivedNeedSuperCall.ts src/services/codefixes/fixExtendsInterfaceBecomesImplements.ts src/services/codefixes/fixForgottenThisPropertyAccess.ts src/services/codefixes/fixSpelling.ts src/services/codefixes/fixes.ts src/services/codefixes/helpers.ts src/services/codefixes/importFixes.ts src/services/codefixes/unusedIdentifierFixes.ts src/services/refactors/convertFunctionToEs6Class.ts src/services/refactors/refactors.ts\n' +
    '\n' +
    'node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types node --pretty --preserveConstEnums --outDir built/local/ --module commonjs -sourcemap --target es5 --lib es6 --noUnusedLocals --noUnusedParameters src/server/cancellationToken/cancellationToken.ts\n' +
    '\n' +
    'node built/local/tsc.js --noImplicitAny --noImplicitThis --noEmitOnError --types node --pretty --preserveConstEnums --outDir built/local/ --out built/local/typingsInstaller.js -sourcemap --target es5 --lib es6 --noUnusedLocals --noUnusedParameters src/server/typingsInstaller/../types.ts src/server/typingsInstaller/../shared.ts src/server/typingsInstaller/typingsInstaller.ts src/server/typingsInstaller/nodeTypingsInstaller.ts\n' +
    '\n' +
    '\n' +
    '\x1B[100;30m125\x1B[0m             process.on("message", (req: DiscoverTypings | CloseProject) => {\n' +
    '\x1B[100;30m   \x1B[0m \x1B[91m                       ~~~~~~~~~\x1B[0m\n' +
    '\n' +
    `src/server/typingsInstaller/nodeTypingsInstaller.ts(125,24): \x1B[91merror\x1B[0m TS2345: Argument of type '"message"' is not assignable to parameter of type 'Signals'.\n`,
  stderr: 'jake aborted.\n' +
    "Error: ENOENT: no such file or directory, unlink 'built/local/typingsInstaller.js'\n" +
    '    at Object.fs.unlinkSync (fs.js:1061:18)\n' +
    '    at Exec.<anonymous> (/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/TypeScript/Jakefile.js:374:16)\n' +
    '(See full trace by running task with --trace)',
  cwd: '/home/david/.nvm/versions/node/v21.5.0/lib/node_modules/every-ts/.data/TypeScript',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}

I think I have the general gist of the problem figured out but I'd be unsure how to fix it. Hopefully this is enough info for you!

[jakebailey]

Thanks for the report! I obviously couldn't test every single commit before publishing it 😄

Just to write down my thoughts, the unlink is a red herring; that path is to delete outputs if another error happens. I suspect that compiler error is the actual problem.

How to fix it... usually this means that the code didn't build at the specific time, even though I ensure that I use a lockfile or npm install --before. So, not totally sure what the fix will be until I play with it.

I think I have the general gist of the problem figured out but I'd be unsure how to fix it.

Just to clarify, what do you have some other thoughts here? Anything helps, unless you aren't meaning that.

[DavidArchibald]

Well I peeked into the source code:

This is the line it errored on: https://github.com/microsoft/TypeScript/blob/3d8cf62846c8bda828675239ddd9d4bd33e53d51/src/server/typingsInstaller/nodeTypingsInstaller.ts#L125

This was its @types/node version: https://github.com/microsoft/TypeScript/blob/3d8cf62846c8bda828675239ddd9d4bd33e53d51/package.json#L47

Now maybe you have some more machinery in place than just trusting the package.json but there's no package-lock.json AND @types/node has the pretty useless version of latest.

[jakebailey]

Now maybe you have some more machinery in place than just trusting the package.json but there's no package-lock.json AND @types/node has the pretty useless version of latest.

I do, in that when there's no lockfile present, I grab the commit timestamp and then run npm install --before <timestamp>, which asks npm to install only packages that would have been available on that timestamp. But, broken commits do happen, so I can try and create a manual workaround.

[DavidArchibald]

Ahhh I see you mentioned npm install --before already. My apologies, I hadn't looked it up before posting my reply so the purpose of the --before flag went over my head.

Testing this specific commit isn't that important to me, you could always maintain a list of not working commits if you want? They'd of course have to be excluded from bisects but it'd be easier than making literally every single commit work. I only happened to stumble onto this commit because of bad luck, just happened to be picked during my bisect.

I would say one improvement is that I had to reinstall every-ts because it seemed like every-ts reset and every-ts switch main wasn't enough to get it to work again; it'd keep trying to build 3d8cf62846c8bda828675239ddd9d4bd33e53d51 even when I did every-ts bisect reset but maybe that's at least partially user error? I just hadn't expected these things to build the current version again so the easiest fix was to just reinstall.

Also I'd be willing to spin up a server to test every revision if you want. It could take several days to run to completion, I'd have to start the process of testing every revision before truly knowing an estimate, but I believe what I'd have to do would be git rev-list + every-ts switch $revision so it'd be pretty cheap to run on the cloud and get an exhaustive list.

[jakebailey]

Testing this specific commit isn't that important to me, you could always maintain a list of not working commits if you want? They'd of course have to be excluded from bisects but it'd be easier than making literally every single commit work.

If one commit fails, it's likely that other commits within the same timespan also fail, so I would try and aim for a more general fix. But, if it comes to it, I can of course figure out which range of commits need a specific version of the node types and check that when building.

Also I'd be willing to spin up a server to test every revision if you want. It could take several days to run to completion, I'd have to start the process of testing every revision before truly knowing an estimate, but I believe what I'd have to do would be git rev-list + every-ts switch $revision so it'd be pretty cheap to run on the cloud and get an exhaustive list.

That would be super interesting, but there are 35k+ commits on the repo... That's like 25 straight days of nonstop building assuming each takes a minute. I don't want to impose that cost on you, of course. Then again, the builds are mostly single threaded, so I bet one could install multiple instances of every-ts and run one per core... Sounds fun 😄

[DavidArchibald]

I've started processing the commits. This'll take several days to weeks at the current rate I'm seeing but it shouldn't cost me anything so don't worry about that.

[DavidArchibald]

Update, I'm on revisions 20,000 out of 35,524.

Currently 872 revisions have failed to build using every-ts, 78 of which seem to hang forever.

[jakebailey]

I'm not surprised that so many don't work; the repo only started merge via squash a few years ago.

[DavidArchibald]

Alright! It's done processing! The tail end was certainly quicker than the beginning.

Here's some information I found interesting:

• Overall 35,524 revisions were built which took a total of 1,121,044 seconds or almost 13 days. A bug in my script did mean I had to recompute some of them and I kept things sequential because it seemed to use enough CPU as was.
• This means the average revision took 31.557... seconds to build.
• 2,270 revisions failed to build, you can see them here: failed.txt
• This means that about 6.39% of all revisions fail to build, though that isn't evenly distributed.
• The longest run of consecutive failures was 144, starting from 6d41964fd097b77abe5ac57e2f7c689cac6300a8 and ending at f45cc4578e80e2ba2de57e4e69005708bcac1590. You can see the runs of failures here: fail-summary.txt.
• 82 revisions seemed to hang forever. I set a limit of 500 seconds on them just to ensure termination. Here's those revisions: hang-forever.txt
• The failed revisions collectively took 203,964 seconds or about 2 days and 8 hours to build.
• This means the average failed revision took 89.85... seconds to build! However this is definitely dragged up by the revisions that took 500 seconds or longer.

I still have the logs and timing information for every individual commit but I haven't uploaded those because they're a lot larger and mostly boring. I'd of course be willing to share them if you'd find that useful. I'm also willing to retest the 2,270 commits iteratively until you bring the number of broken commits down. HOWEVER I fully understand how much manual labor it could take to deal with these 2,270 commits.

A simple approach to making every-ts more durable to these commits that fail could be to blacklist all of these commits. That would entail stopping a user from manually switching to these commits and then during every-ts bisect use git bisect skip using the range notation to tell it to skip these commits that don't build.

[jakebailey]

Awesome! I really appreciate this data. I have some hope that I can fix some of these, but I suspect a bunch are just WIP commits from long ago (which a bisect with --first-parent would skip).

I can certainly include a list of all broken commits and error if any are checked out, though I'd like to take a crack at them regardless.

[jakebailey]

I still have the logs and timing information for every individual commit but I haven't uploaded those because they're a lot larger and mostly boring. I'd of course be willing to share them if you'd find that useful.

I actually think they might be useful; if I'm going to fix certain classes of historical errors, it'd be nice to get them in buckets. E.g. the original report breaks due to:

src/server/typingsInstaller/nodeTypingsInstaller.ts(125,24): error TS2345: Argument of type '"message"' is not assignable to parameter of type 'Signals'.

So is a place where the repo was actually broken, and I could apply a patch for this kind of error by any-ing something out. But, I don't know how many commits have this same error to know if it's worth it.

[jakebailey]

I also checked some of the ones that hang, and the first one I checked (34b9c4dbad569a21227e8761f785630c6052a855) didn't; but it did have some compiler error that showed that it was sensitive to types that were installed within the node_modules dir where every-ts was, which is another reason to do #1.