microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
101.3k stars 12.53k forks source link

โšก Performance: Project service doesn't cache all fs.statSync #59338

Open JoshuaKGoldberg opened 4 months ago

JoshuaKGoldberg commented 4 months ago

Acknowledgement

Comment

๐Ÿ”Ž Search Terms

project service fs stat statSync

๐Ÿ™ Actual behavior

When using typescript-eslint's parserOptions.projectService, type checking APIs switch from the traditional manual TypeScript ts.Program approach to the editor-style ts.ProjectService. We're observing excess calls to the ts.sys.statSync function on some paths in node_modules/ - up to a few dozen for some paths (!).

๐Ÿ™‚ Expected behavior

There should be no uncached statSync calls, I'd think? Even if in a persistent session, I'd expect them to be debounced in some way.

As a draft, I added a basic caching Map to statSync and ran a before & after comparison with hyperfine. The results showed a ~7-12% improvement in lint time:

Variant Measurement User Time
Baseline 3.112 s ยฑ 0.033 s 4.382
Caching 2.740 s ยฑ 0.030 s 4.032
diff patch to switch to the Caching variant... ```diff diff --git a/node_modules/typescript/lib/typescript.js b/node_modules/typescript/lib/typescript.js index 4baad59..44639d5 100644 --- a/node_modules/typescript/lib/typescript.js +++ b/node_modules/typescript/lib/typescript.js @@ -8546,9 +8546,15 @@ var sys = (() => { } } }; + const statCache = new Map(); return nodeSystem; function statSync(path) { - return _fs.statSync(path, { throwIfNoEntry: false }); + if (statCache.has(path)) { + return statCache.get(path); + } + const result = _fs.statSync(path, { throwIfNoEntry: false }); + statCache.set(path, result); + return result; } function enableCPUProfiler(path, cb) { if (activeSession) { ```

Additional information about the issue

On the typescript-eslint side:

These are the top 10 most common paths called by statSync... ```plaintext 32 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments 28 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio 28 Error: /Users/josh/repos/performance/node_modules/execa/types/return 26 Error: /Users/josh/repos/performance/node_modules/execa/types 17 Error: /Users/josh/repos/performance/node_modules/execa/types/methods 14 Error: /Users/josh/repos/performance/node_modules/execa/types/subprocess 9 Error: /Users/josh/repos/performance/node_modules/prettier 9 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments/options.d.ts 8 Error: /Users/josh/repos/performance/node_modules/execa/types/transform 7 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio/type.d.ts ```
Here's an example call stack from the most common one... ```plaintext Error at statSync (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8568:19) at fileSystemEntryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8794:22) at Object.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8816:14) at _AutoImportProviderProject.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182483:40) at directoryProbablyExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:20914:40) at tryAddingExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44501:29) at loadModuleFromFileNoImplicitExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44484:10) at loadModuleFromFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44460:40) at nodeLoadModuleByRelativeName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44409:30) at tryResolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44365:25) at nodeModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44283:14) at nodeNextModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44166:10) at nodeNextModuleNameResolver (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44148:10) at resolveModuleName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:43972:18) at resolveModuleNameUsingGlobalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129141:25) at Object.resolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129128:45) at resolveNamesWithLocalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129434:29) at Object.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129523:12) at _AutoImportProviderProject.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182461:33) at resolveModuleNamesWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124514:20) at resolveModuleNamesReusingOldState (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124600:14) at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126117:118) at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7) at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20) at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126143:11) at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7) at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20) at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125695:22 at getSourceFileFromReferenceWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125666:26) at processSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125693:5) at processRootFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125485:5) at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:41 at forEach (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:2387:22) at createProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:5) at synchronizeHostDataWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148957:15) at synchronizeHostData (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148853:7) at Object.getProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:149029:5) at _AutoImportProviderProject.updateGraphWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183153:41) at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183009:32) at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184155:37) at updateProjectIfDirty (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184785:36) at ConfiguredProject2.getPackageJsonAutoImportProvider (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183741:9) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183033:12) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184311:24) at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11) at _ProjectService.reloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:186410:5) at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184307:29) at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11) at updateConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184802:9) at _ProjectService.findCreateOrReloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187271:44) at _ProjectService.tryFindDefaultConfiguredProjectForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187295:25) at _ProjectService.tryFindDefaultConfiguredProjectAndLoadAncestorsForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187341:25) at _ProjectService.assignProjectToOpenedScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187226:27) at _ProjectService.openClientFileWithNormalizedPath (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187421:48) at _ProjectService.openClientFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187129:17) at useProgramFromProjectService (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/useProgramFromProjectService.js:61:28) at getProgramAndAST (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:44:100) at parseAndGenerateServices (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:155:11) at Object.parseForESLint (/Users/josh/repos/performance/node_modules/@typescript-eslint/parser/dist/parser.js:101:80) at Object.parse (/Users/josh/repos/performance/node_modules/eslint/lib/languages/js/index.js:186:26) at parse (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:931:29) at Linter._verifyWithFlatConfigArrayAndWithoutProcessors (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1696:33) at Linter._verifyWithFlatConfigArray (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2062:21) at Linter.verify (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1528:61) at Linter.verifyAndFix (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2299:29) at verifyText (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:498:48) at /Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:939:40 at async Promise.all (index 1) at async ESLint.lintFiles (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:880:25) at async Object.execute (/Users/josh/repos/performance/node_modules/eslint/lib/cli.js:521:23) at async main (/Users/josh/repos/performance/node_modules/eslint/bin/eslint.js:153:22) ```

cc @sheetalkamat as FYI, after a pairing with @jakebailey.

sheetalkamat commented 4 months ago

I will take a look but do you really need AutoImportProviderProject ? If not you should probably set the options to disable that ? Currently auto import provider cannot reuse the resolutions from configured project (i am looking into sharing those as part of #55968 and may be that will help but i will investigate the stacks soon.

jakebailey commented 4 months ago

What options should they be setting to make that project not used? They definitely don't need auto import anything.

sheetalkamat commented 4 months ago

preferences.includePackageJsonAutoImports need to be set with setHostConfiguration

sheetalkamat commented 4 months ago

@jakebailey can you try and see if un cached statsync and realpath calls reduce after that?

jakebailey commented 4 months ago

Hm, I sent https://github.com/typescript-eslint/typescript-eslint/pull/9586 but it's saying:

    Received message: "service.setHostConfiguration is not a function"
          68 |   });
          69 |
        > 70 |   service.setHostConfiguration({
             |           ^
          71 |     preferences: {
          72 |       includePackageJsonAutoImports: 'off',
          73 |     },
          at setHostConfiguration (src/create-program/createProjectService.ts:70:11)
          at tests/lib/createProjectService.test.ts:53:27

Have not tested locally (will do that shortly); maybe this func is new?

sheetalkamat commented 4 months ago

Seems like atleast 5 year old

JoshuaKGoldberg commented 4 months ago

Oh, the unit tests in typescript-eslint mock out the service - see https://github.com/typescript-eslint/typescript-eslint/blob/5542aeb9440280dd4c0be529e9842f12221737ff/packages/typescript-estree/tests/lib/createProjectService.test.ts#L8

jakebailey commented 4 months ago

Ah, yeah, so I'll definitely have to real-world test it.

sheetalkamat commented 4 months ago

I think the eslint change to disable auto import provider should help with the stats cache issue. With that change those high number of stats cache hitting goes down significantly. I confirmed that by running tsserver that just by default returns "off" for that option.

The other two locations where we get from cache are:

jakebailey commented 4 months ago

Before:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ files โ”‚ project (even)       โ”‚ project (references) โ”‚ service (even)       โ”‚ service (references) โ”‚
โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 128   โ”‚ '1.788 s ยฑ  0.040 s' โ”‚ '1.788 s ยฑ  0.041 s' โ”‚ '1.861 s ยฑ  0.053 s' โ”‚ '2.721 s ยฑ  0.040 s' โ”‚
โ”‚ 512   โ”‚ '2.747 s ยฑ  0.032 s' โ”‚ '2.766 s ยฑ  0.042 s' โ”‚ '2.974 s ยฑ  0.036 s' โ”‚ '4.032 s ยฑ  0.041 s' โ”‚
โ”‚ 1024  โ”‚ '3.943 s ยฑ  0.044 s' โ”‚ '4.008 s ยฑ  0.040 s' โ”‚ '4.644 s ยฑ  0.053 s' โ”‚ '5.925 s ยฑ  0.180 s' โ”‚
โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

After https://github.com/typescript-eslint/typescript-eslint/pull/9586:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ files โ”‚ project (even)       โ”‚ project (references) โ”‚ service (even)       โ”‚ service (references) โ”‚
โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 128   โ”‚ '1.753 s ยฑ  0.019 s' โ”‚ '1.761 s ยฑ  0.019 s' โ”‚ '1.804 s ยฑ  0.036 s' โ”‚ '2.478 s ยฑ  0.046 s' โ”‚
โ”‚ 512   โ”‚ '2.697 s ยฑ  0.049 s' โ”‚ '2.707 s ยฑ  0.031 s' โ”‚ '2.936 s ยฑ  0.039 s' โ”‚ '3.837 s ยฑ  0.050 s' โ”‚
โ”‚ 1024  โ”‚ '3.880 s ยฑ  0.044 s' โ”‚ '3.916 s ยฑ  0.028 s' โ”‚ '4.511 s ยฑ  0.030 s' โ”‚ '5.493 s ยฑ  0.065 s' โ”‚
โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

So, clearly better by a little bit.

I haven't yet done the same stat counting augmentation; I'm not sure which command to run to exactly reproduce what @JoshuaKGoldberg was doing anymore.

jakebailey commented 4 months ago

The test was just me in the linked performance repo, modifying node_modules with:

image

jakebailey commented 4 months ago

Okay, I checked and setting includePackageJsonAutoImports: 'off' does indeed reduce stat down to one per file, great!

jakebailey commented 4 months ago

Hm, then I remove the code that sets that and the result is the same; I may just be measuring this wrong. I think I'd need Josh to retest.

JoshuaKGoldberg commented 4 months ago

I see it reducing the countsย (hooray!) down to three per file. Am I holding it wrong, maybe?

...
3 /Users/josh/repos/performance/node_modules/undici-types/pool-stats.d.ts
3 /Users/josh/repos/performance/node_modules/undici-types/patch.d.ts
3 /Users/josh/repos/performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts

This from:

  1. Entering cases/files-1024-layout-even-singlerun-false-types-service from github.com/typescript-eslint/performance
  2. Applying the change in https://github.com/typescript-eslint/typescript-eslint/pull/9586 (but outside the if (options.defaultProject) {
  3. Changing statSync in node_modules/typescript/lib/typescript.js like:

    const statCounts = new Map();
    
    setTimeout(() => {
      console.log(
        Array.from(statCounts)
          .sort(([, a], [, b]) => a - b)
          .map((pair) => pair.reverse().join(" "))
          .join("\n")
      )
    }, 2500)
    
    return nodeSystem;
    
    function statSync(path) {
      statCounts.set(path, (statCounts.get(path) || 0) + 1);
    
      return _fs.statSync(path, { throwIfNoEntry: false });
    }
jakebailey commented 4 months ago

cases/files-1024-layout-even-singlerun-false-types-service doesn't exist for me, but using cases/files-1024-layout-even-singlerun-true-types-service, before I see:

9 /home/jabaile/work/TypeScript-eslint-performance/node_modules/prettier
9 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/arguments/options.d.ts
14 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/subprocess
17 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/methods
26 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types
28 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/stdio
28 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/return
32 /home/jabaile/work/TypeScript-eslint-performance/node_modules/execa/types/arguments

Then after I see:

3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/interceptors.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/header.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/readable.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/pool-stats.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/undici-types/patch.d.ts
3 /home/jabaile/work/TypeScript-eslint-performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts

So, yeah, there's something odd going on here.

sheetalkamat commented 4 months ago

For files in node modules it will be 3 instead of 2 (i forgot to add that info in my post) because its used to get modified time while watching as we would be watching node_modules folder and thats how we determine file change. This is my yet another wip and incomplete experiment where NodeSystem should be able to share the stats apart from caching fileExists etc calls that we already do. But I determined it not to be priority based on this exact same analysis that the max count we would hit the cache is 2 ( + 1 initial) in project service and 1 (+1 initial) in tsbuild scneario. This needs bigger API level change that would be tricky to do and make sure API users are using it correctly etc. (involves invalidating part that can be tricky)

jakebailey commented 4 months ago

Here are the three traces for a file, as an example:

/home/jabaile/work/TypeScript-eslint-performance/node_modules/@types/mdast/node_modules/@types/unist/index.d.ts 3 Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at fileSystemEntryExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8785:22)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8804:14)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:122814:130)
    at ConfiguredProject2.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182448:76)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148882:38)
    at host.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:123387:41)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:127085:28)
    at tryFileLookup (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:44534:20)
    at tryFile (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:44525:12)
Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at fileSystemEntryExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8785:22)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8804:14)
    at Object.fileExists (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:122814:130)
    at _ProjectService.getOrCreateScriptInfoWorker (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186740:84)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClientForNormalizedPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186696:19)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClient (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186511:17)
    at ConfiguredProject2.getOrCreateScriptInfoAndAttachToProject (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182388:44)
    at ConfiguredProject2.getScriptSnapshot (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182419:29)
    at getOrCreateSourceFileByPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148995:35)
Error
    at statSync (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8564:64)
    at Object.getModifiedTime3 [as getModifiedTime] (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:8827:22)
    at _ProjectService.getModifiedTime (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186670:23)
    at _ProjectService.watchClosedScriptInfo (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186589:27)
    at _ProjectService.getOrCreateScriptInfoWorker (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186747:14)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClientForNormalizedPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186696:19)
    at _ProjectService.getOrCreateScriptInfoNotOpenedByClient (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:186511:17)
    at ConfiguredProject2.getOrCreateScriptInfoAndAttachToProject (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182388:44)
    at ConfiguredProject2.getScriptSnapshot (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:182419:29)
    at getOrCreateSourceFileByPath (/home/jabaile/work/TypeScript-eslint-performance/node_modules/typescript/lib/typescript.js:148995:35)
jakebailey commented 4 months ago

Now, what might be possible for ts-eslint is when you infer that it's a single run, provide a host that caches everything, since you're assuming that it's just going to exit after running anyway.