microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
100.07k stars 12.37k forks source link

โšก Performance: Project service doesn't cache all fs.statSync #59338

Open JoshuaKGoldberg opened 1 month ago

JoshuaKGoldberg commented 1 month ago

Acknowledgement

Comment

๐Ÿ”Ž Search Terms

project service fs stat statSync

๐Ÿ™ Actual behavior

When using typescript-eslint's parserOptions.projectService, type checking APIs switch from the traditional manual TypeScript ts.Program approach to the editor-style ts.ProjectService. We're observing excess calls to the ts.sys.statSync function on some paths in node_modules/ - up to a few dozen for some paths (!).

๐Ÿ™‚ Expected behavior

There should be no uncached statSync calls, I'd think? Even if in a persistent session, I'd expect them to be debounced in some way.

As a draft, I added a basic caching Map to statSync and ran a before & after comparison with hyperfine. The results showed a ~7-12% improvement in lint time:

Variant Measurement User Time
Baseline 3.112 s ยฑ 0.033 s 4.382
Caching 2.740 s ยฑ 0.030 s 4.032
diff patch to switch to the Caching variant... ```diff diff --git a/node_modules/typescript/lib/typescript.js b/node_modules/typescript/lib/typescript.js index 4baad59..44639d5 100644 --- a/node_modules/typescript/lib/typescript.js +++ b/node_modules/typescript/lib/typescript.js @@ -8546,9 +8546,15 @@ var sys = (() => { } } }; + const statCache = new Map(); return nodeSystem; function statSync(path) { - return _fs.statSync(path, { throwIfNoEntry: false }); + if (statCache.has(path)) { + return statCache.get(path); + } + const result = _fs.statSync(path, { throwIfNoEntry: false }); + statCache.set(path, result); + return result; } function enableCPUProfiler(path, cb) { if (activeSession) { ```

Additional information about the issue

On the typescript-eslint side:

These are the top 10 most common paths called by statSync... ```plaintext 32 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments 28 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio 28 Error: /Users/josh/repos/performance/node_modules/execa/types/return 26 Error: /Users/josh/repos/performance/node_modules/execa/types 17 Error: /Users/josh/repos/performance/node_modules/execa/types/methods 14 Error: /Users/josh/repos/performance/node_modules/execa/types/subprocess 9 Error: /Users/josh/repos/performance/node_modules/prettier 9 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments/options.d.ts 8 Error: /Users/josh/repos/performance/node_modules/execa/types/transform 7 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio/type.d.ts ```
Here's an example call stack from the most common one... ```plaintext Error at statSync (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8568:19) at fileSystemEntryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8794:22) at Object.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8816:14) at _AutoImportProviderProject.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182483:40) at directoryProbablyExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:20914:40) at tryAddingExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44501:29) at loadModuleFromFileNoImplicitExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44484:10) at loadModuleFromFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44460:40) at nodeLoadModuleByRelativeName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44409:30) at tryResolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44365:25) at nodeModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44283:14) at nodeNextModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44166:10) at nodeNextModuleNameResolver (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44148:10) at resolveModuleName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:43972:18) at resolveModuleNameUsingGlobalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129141:25) at Object.resolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129128:45) at resolveNamesWithLocalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129434:29) at Object.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129523:12) at _AutoImportProviderProject.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182461:33) at resolveModuleNamesWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124514:20) at resolveModuleNamesReusingOldState (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124600:14) at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126117:118) at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7) at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20) at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126143:11) at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7) at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20) at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125695:22 at getSourceFileFromReferenceWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125666:26) at processSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125693:5) at processRootFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125485:5) at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:41 at forEach (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:2387:22) at createProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:5) at synchronizeHostDataWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148957:15) at synchronizeHostData (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148853:7) at Object.getProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:149029:5) at _AutoImportProviderProject.updateGraphWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183153:41) at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183009:32) at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184155:37) at updateProjectIfDirty (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184785:36) at ConfiguredProject2.getPackageJsonAutoImportProvider (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183741:9) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183033:12) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184311:24) at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11) at _ProjectService.reloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:186410:5) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184307:29) at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11) at updateConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184802:9) at _ProjectService.findCreateOrReloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187271:44) at _ProjectService.tryFindDefaultConfiguredProjectForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187295:25) at _ProjectService.tryFindDefaultConfiguredProjectAndLoadAncestorsForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187341:25) at _ProjectService.assignProjectToOpenedScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187226:27) at _ProjectService.openClientFileWithNormalizedPath (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187421:48) at _ProjectService.openClientFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187129:17) at useProgramFromProjectService (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/useProgramFromProjectService.js:61:28) at getProgramAndAST (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:44:100) at parseAndGenerateServices (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:155:11) at Object.parseForESLint (/Users/josh/repos/performance/node_modules/@typescript-eslint/parser/dist/parser.js:101:80) at Object.parse (/Users/josh/repos/performance/node_modules/eslint/lib/languages/js/index.js:186:26) at parse (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:931:29) at Linter._verifyWithFlatConfigArrayAndWithoutProcessors (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1696:33) at Linter._verifyWithFlatConfigArray (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2062:21) at Linter.verify (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1528:61) at Linter.verifyAndFix (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2299:29) at verifyText (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:498:48) at /Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:939:40 at async Promise.all (index 1) at async ESLint.lintFiles (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:880:25) at async Object.execute (/Users/josh/repos/performance/node_modules/eslint/lib/cli.js:521:23) at async main (/Users/josh/repos/performance/node_modules/eslint/bin/eslint.js:153:22) ```

cc @sheetalkamat as FYI, after a pairing with @jakebailey.

sheetalkamat commented 1 month ago

I will take a look but do you really need AutoImportProviderProject ? If not you should probably set the options to disable that ? Currently auto import provider cannot reuse the resolutions from configured project (i am looking into sharing those as part of #55968 and may be that will help but i will investigate the stacks soon.

jakebailey commented 1 month ago

What options should they be setting to make that project not used? They definitely don't need auto import anything.

sheetalkamat commented 1 month ago

preferences.includePackageJsonAutoImports need to be set with setHostConfiguration

sheetalkamat commented 1 month ago

@jakebailey can you try and see if un cached statsync and realpath calls reduce after that?

jakebailey commented 1 month ago

Hm, I sent https://github.com/typescript-eslint/typescript-eslint/pull/9586 but it's saying:

    Received message: "service.setHostConfiguration is not a function"
          68 |   });
          69 |
        > 70 |   service.setHostConfiguration({
             |           ^
          71 |     preferences: {
          72 |       includePackageJsonAutoImports: 'off',
          73 |     },
          at setHostConfiguration (src/create-program/createProjectService.ts:70:11)
          at tests/lib/createProjectService.test.ts:53:27

Have not tested locally (will do that shortly); maybe this func is new?

sheetalkamat commented 1 month ago

Seems like atleast 5 year old

JoshuaKGoldberg commented 1 month ago

Oh, the unit tests in typescript-eslint mock out the service - see https://github.com/typescript-eslint/typescript-eslint/blob/5542aeb9440280dd4c0be529e9842f12221737ff/packages/typescript-estree/tests/lib/createProjectService.test.ts#L8

jakebailey commented 1 month ago

Ah, yeah, so I'll definitely have to real-world test it.

sheetalkamat commented 1 month ago

I think the eslint change to disable auto import provider should help with the stats cache issue. With that change those high number of stats cache hitting goes down significantly. I confirmed that by running tsserver that just by default returns "off" for that option.

The other two locations where we get from cache are:

jakebailey commented 1 month ago

Before:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ files โ”‚ project (even)       โ”‚ project (references) โ”‚ service (even)       โ”‚ service (references) โ”‚
โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 128   โ”‚ '1.788 s ยฑ  0.040 s' โ”‚ '1.788 s ยฑ  0.041 s' โ”‚ '1.861 s ยฑ  0.053 s' โ”‚ '2.721 s ยฑ  0.040 s' โ”‚
โ”‚ 512   โ”‚ '2.747 s ยฑ  0.032 s' โ”‚ '2.766 s ยฑ  0.042 s' โ”‚ '2.974 s ยฑ  0.036 s' โ”‚ '4.032 s ยฑ  0.041 s' โ”‚
โ”‚ 1024  โ”‚ '3.943 s ยฑ  0.044 s' โ”‚ '4.008 s ยฑ  0.040 s' โ”‚ '4.644 s ยฑ  0.053 s' โ”‚ '5.925 s ยฑ  0.180 s' โ”‚
โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

After https://github.com/typescript-eslint/typescript-eslint/pull/9586:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ files โ”‚ project (even)       โ”‚ project (references) โ”‚ service (even)       โ”‚ service (references) โ”‚
โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 128   โ”‚ '1.753 s ยฑ  0.019 s' โ”‚ '1.761 s ยฑ  0.019 s' โ”‚ '1.804 s ยฑ  0.036 s' โ”‚ '2.478 s ยฑ  0.046 s' โ”‚
โ”‚ 512   โ”‚ '2.697 s ยฑ  0.049 s' โ”‚ '2.707 s ยฑ  0.031 s' โ”‚ '2.936 s ยฑ  0.039 s' โ”‚ '3.837 s ยฑ  0.050 s' โ”‚
โ”‚ 1024  โ”‚ '3.880 s ยฑ  0.044 s' โ”‚ '3.916 s ยฑ  0.028 s' โ”‚ '4.511 s ยฑ  0.030 s' โ”‚ '5.493 s ยฑ  0.065 s' โ”‚
โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

So, clearly better by a little bit.

I haven't yet done the same stat counting augmentation; I'm not sure which command to run to exactly reproduce what @JoshuaKGoldberg was doing anymore.

jakebailey commented 1 month ago

The test was just me in the linked performance repo, modifying node_modules with:

image

jakebailey commented 1 month ago

Okay, I checked and setting includePackageJsonAutoImports: 'off' does indeed reduce stat down to one per file, great!

jakebailey commented 1 month ago

Hm, then I remove the code that sets that and the result is the same; I may just be measuring this wrong. I think I'd need Josh to retest.

JoshuaKGoldberg commented 1 month ago

I see it reducing the countsย (hooray!) down to three per file. Am I holding it wrong, maybe?

...
3 /Users/josh/repos/performance/node_modules/undici-types/pool-stats.d.ts
3 /Users/josh/repos/performance/node_modules/undici-types/patch.d.ts
3 /Users/josh/repos/performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts

This from:

  1. Entering cases/files-1024-layout-even-singlerun-false-types-service from github.com/typescript-eslint/performance
  2. Applying the change in https://github.com/typescript-eslint/typescript-eslint/pull/9586 (but outside the if (options.defaultProject) {
  3. Changing statSync in node_modules/typescript/lib/typescript.js like:

    const statCounts = new Map();
    
    setTimeout(() => {
      console.log(
        Array.from(statCounts)
          .sort(([, a], [, b]) => a - b)
          .map((pair) => pair.reverse().join(" "))
          .join("\n")
      )
    }, 2500)
    
    return nodeSystem;
    
    function statSync(path) {
      statCounts.set(path, (statCounts.get(path) || 0) + 1);
    
      return _fs.statSync(path, { throwIfNoEntry: false });
    }
jakebailey commented 1 month ago

cases/files-1024-layout-even-singlerun-false-types-service doesn't exist for me, but using cases/files-1024-layout-even-singlerun-true-types-service, before I see:

9 /home/jabaile/work/TypeScript-eslint-performance/node_modules/prettier
9 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/arguments/options.d.ts
14 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/subprocess
17 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/methods
26 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types
28 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/stdio
28 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/return
32 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/arguments

Then after I see:

3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/interceptors.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/header.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/readable.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/pool-stats.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/patch.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts

So, yeah, there's something odd going on here.

sheetalkamat commented 1 month ago

For files in node modules it will be 3 instead of 2 (i forgot to add that info in my post) because its used to get modified time while watching as we would be watching node_modules folder and thats how we determine file change. This is my yet another wip and incomplete experiment where NodeSystem should be able to share the stats apart from caching fileExists etc calls that we already do. But I determined it not to be priority based on this exact same analysis that the max count we would hit the cache is 2 ( + 1 initial) in project service and 1 (+1 initial) in tsbuild scneario. This needs bigger API level change that would be tricky to do and make sure API users are using it correctly etc. (involves invalidating part that can be tricky)

jakebailey commented 1 month ago

Here are the three traces for a file, as an example:

/home/jabaile/work/TypeScript-eslint-performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts 3 Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at fileSystemEntryExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8785:22)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8804:14)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:122814:130)
    at ConfiguredProject2.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182448:76)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148882:38)
    at host.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:123387:41)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:127085:28)
    at tryFileLookup (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:44534:20)
    at tryFile (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:44525:12)
Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at fileSystemEntryExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8785:22)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8804:14)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:122814:130)
    at _ProjectService.getOrCreateScriptInfoWorker (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186740:84)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClientForNormalizedPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186696:19)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClient (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186511:17)
    at ConfiguredProject2.getOrCreateScriptInfoAndAttachToProject (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182388:44)
    at ConfiguredProject2.getScriptSnapshot (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182419:29)
    at getOrCreateSourceFileByPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148995:35)
Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at Object.getModifiedTime3 [as getModifiedTime] (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8827:22)
    at _ProjectService.getModifiedTime (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186670:23)
    at _ProjectService.watchClosedScriptInfo (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186589:27)
    at _ProjectService.getOrCreateScriptInfoWorker (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186747:14)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClientForNormalizedPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186696:19)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClient (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186511:17)
    at ConfiguredProject2.getOrCreateScriptInfoAndAttachToProject (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182388:44)
    at ConfiguredProject2.getScriptSnapshot (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182419:29)
    at getOrCreateSourceFileByPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148995:35)
jakebailey commented 1 month ago

Now, what might be possible for ts-eslint is when you infer that it's a single run, provide a host that caches everything, since you're assuming that it's just going to exit after running anyway.