search

microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
98.27k stars 12.21k forks source link

Improve diagnostics deduplication 2 #58318

Closed gabritto closed 2 weeks ago

gabritto commented 2 weeks ago

Fixes #58207. Follow-up to #58220.

This applies deduplication for the two scenarios I'm aware of where we can produce different diagnostics (with different message and code) that report the same problem at the same location, depending on the current state of the type checker: (1) not found diagnostics that may or may not have suggestions, and (2) type comparison diagnostics that may or may not suppress the top part of a diagnostic chain depending on whether we have a more useful elaboration or not.

Case (1) depends on the state of the checker because we have a constant limit on how many times we try to look for suggestion, for performance reasons. Case (2) depends on the state of the checker because we don't provide a more detailed elaboration when we already have a cached result for the type comparison.

The general idea here is to make these different diagnostics that refer to the same problem point to the same canonical diagnostic, and this canonical diagnostic is used for comparison and deciding equivalence. e.g. in case (1), we can produce both "Cannot find name 'x'." and "Cannot find name 'x'. Did you mean 'y'?", so now the diagnostic with message "Cannot find name 'x'. Did you mean 'y'?" will have a canonical diagnostic of "Cannot find name 'x'.", and will be considered equivalent to "Cannot find name 'x'." (but the diagnostic with the suggestion will be preferred).

gabritto commented 2 weeks ago

@typescript-bot test it

typescript-bot commented 2 weeks ago

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
test top400 ✅ Started ✅ Results
user test this ✅ Started ✅ Results
run dt ✅ Started ✅ Results
perf test this faster ✅ Started 👀 Results
typescript-bot commented 2 weeks ago

Hey @gabritto, the results of running the DT tests are ready.

Everything looks the same!

You can check the log here.

typescript-bot commented 2 weeks ago

@gabritto Here are the results of running the user tests comparing main and refs/pull/58318/merge:

Everything looks good!

typescript-bot commented 2 weeks ago

@gabritto The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,154 62,154 ~ ~ ~ p=1.000 n=6
Types 50,273 50,273 ~ ~ ~ p=1.000 n=6
Memory used 193,385k (± 0.97%) 192,935k (± 0.73%) ~ 192,128k 195,786k p=0.936 n=6
Parse Time 1.35s (± 2.01%) 1.35s (± 1.30%) ~ 1.32s 1.36s p=1.000 n=6
Bind Time 0.72s (± 0.57%) 0.72s ~ ~ ~ p=0.405 n=6
Check Time 9.56s (± 0.41%) 9.60s (± 0.53%) ~ 9.55s 9.69s p=0.257 n=6
Emit Time 2.62s (± 0.62%) 2.62s (± 0.52%) ~ 2.60s 2.64s p=0.512 n=6
Total Time 14.25s (± 0.22%) 14.28s (± 0.29%) ~ 14.24s 14.35s p=0.295 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 945,172 945,172 ~ ~ ~ p=1.000 n=6
Types 408,068 408,068 ~ ~ ~ p=1.000 n=6
Memory used 1,221,995k (± 0.00%) 1,222,000k (± 0.00%) ~ 1,221,935k 1,222,047k p=0.689 n=6
Parse Time 6.92s (± 0.66%) 6.91s (± 0.35%) ~ 6.88s 6.95s p=0.686 n=6
Bind Time 1.87s (± 0.74%) 1.87s (± 0.73%) ~ 1.84s 1.88s p=0.933 n=6
Check Time 31.39s (± 0.16%) 31.32s (± 0.42%) ~ 31.13s 31.47s p=0.520 n=6
Emit Time 14.64s (± 0.67%) 14.66s (± 0.46%) ~ 14.53s 14.72s p=0.229 n=6
Total Time 54.83s (± 0.23%) 54.76s (± 0.33%) ~ 54.44s 54.92s p=0.748 n=6
mui-docs - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 1,954,598 1,954,598 ~ ~ ~ p=1.000 n=6
Types 676,387 676,387 ~ ~ ~ p=1.000 n=6
Memory used 1,753,330k (± 0.00%) 1,753,397k (± 0.00%) +67k (+ 0.00%) 1,753,348k 1,753,416k p=0.008 n=6
Parse Time 6.88s (± 0.17%) 6.89s (± 0.16%) ~ 6.88s 6.91s p=0.244 n=6
Bind Time 2.30s (± 0.27%) 2.30s (± 0.33%) ~ 2.29s 2.31s p=0.718 n=6
Check Time 56.78s (± 0.51%) 56.96s (± 0.44%) ~ 56.63s 57.29s p=0.261 n=6
Emit Time 0.14s (± 2.95%) 0.14s ~ ~ ~ p=0.405 n=6
Total Time 66.10s (± 0.44%) 66.29s (± 0.36%) ~ 65.97s 66.60s p=0.173 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,215,071 1,215,421 +350 (+ 0.03%) ~ ~ p=0.001 n=6
Types 257,359 257,437 +78 (+ 0.03%) ~ ~ p=0.001 n=6
Memory used 2,322,684k (± 0.03%) 2,324,520k (± 0.02%) +1,836k (+ 0.08%) 2,323,931k 2,325,243k p=0.005 n=6
Parse Time 5.06s (± 0.86%) 5.07s (± 0.77%) ~ 5.02s 5.13s p=0.873 n=6
Bind Time 1.86s (± 0.66%) 1.89s (± 0.80%) +0.02s (+ 1.07%) 1.86s 1.90s p=0.048 n=6
Check Time 33.94s (± 0.29%) 33.84s (± 0.27%) ~ 33.71s 33.96s p=0.149 n=6
Emit Time 2.71s (± 2.09%) 2.65s (± 2.10%) ~ 2.59s 2.74s p=0.078 n=6
Total Time 43.61s (± 0.33%) 43.46s (± 0.20%) ~ 43.30s 43.56s p=0.128 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,215,071 1,215,421 +350 (+ 0.03%) ~ ~ p=0.001 n=6
Types 257,359 257,437 +78 (+ 0.03%) ~ ~ p=0.001 n=6
Memory used 2,397,952k (± 0.04%) 2,398,954k (± 0.03%) ~ 2,398,073k 2,399,698k p=0.066 n=6
Parse Time 6.32s (± 0.92%) 6.33s (± 0.78%) ~ 6.27s 6.41s p=0.748 n=6
Bind Time 2.00s (± 1.38%) 2.03s (± 1.20%) ~ 1.99s 2.06s p=0.146 n=6
Check Time 40.49s (± 0.44%) 40.58s (± 0.33%) ~ 40.41s 40.72s p=0.575 n=6
Emit Time 3.27s (± 2.71%) 3.26s (± 2.68%) ~ 3.13s 3.34s p=0.810 n=6
Total Time 52.09s (± 0.27%) 52.21s (± 0.46%) ~ 51.96s 52.53s p=0.471 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 256,220 256,295 +75 (+ 0.03%) ~ ~ p=0.001 n=6
Types 103,640 103,661 +21 (+ 0.02%) ~ ~ p=0.001 n=6
Memory used 424,115k (± 0.01%) 424,284k (± 0.01%) +169k (+ 0.04%) 424,238k 424,361k p=0.005 n=6
Parse Time 3.49s (± 1.19%) 3.51s (± 0.88%) ~ 3.46s 3.55s p=0.629 n=6
Bind Time 1.31s (± 0.79%) 1.33s (± 3.02%) ~ 1.31s 1.41s p=0.210 n=6
Check Time 18.21s (± 0.37%) 18.26s (± 0.30%) ~ 18.19s 18.32s p=0.127 n=6
Emit Time 1.40s (± 0.84%) 1.37s (± 1.87%) -0.03s (- 2.15%) 1.33s 1.41s p=0.042 n=6
Total Time 24.40s (± 0.39%) 24.47s (± 0.33%) ~ 24.40s 24.62s p=0.420 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,824 224,824 ~ ~ ~ p=1.000 n=6
Types 93,390 93,390 ~ ~ ~ p=1.000 n=6
Memory used 369,284k (± 0.01%) 369,337k (± 0.02%) ~ 369,285k 369,456k p=0.108 n=6
Parse Time 3.67s (± 0.53%) 3.68s (± 0.64%) ~ 3.65s 3.71s p=0.568 n=6
Bind Time 1.94s (± 0.69%) 1.94s (± 0.96%) ~ 1.91s 1.96s p=0.935 n=6
Check Time 19.44s (± 0.37%) 19.46s (± 0.18%) ~ 19.41s 19.50s p=0.936 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 25.05s (± 0.27%) 25.07s (± 0.23%) ~ 25.00s 25.16s p=0.575 n=6
vscode - node (v18.15.0, x64)
Errors 4 4 ~ ~ ~ p=1.000 n=6
Symbols 2,795,894 2,795,894 ~ ~ ~ p=1.000 n=6
Types 949,958 949,958 ~ ~ ~ p=1.000 n=6
Memory used 2,924,603k (± 0.01%) 2,924,669k (± 0.00%) ~ 2,924,526k 2,924,783k p=0.378 n=6
Parse Time 13.44s (± 0.58%) 13.43s (± 0.28%) ~ 13.38s 13.46s p=0.808 n=6
Bind Time 4.04s (± 0.31%) 4.08s (± 1.80%) ~ 4.04s 4.23s p=0.087 n=6
Check Time 72.24s (± 0.46%) 72.70s (± 0.66%) +0.46s (+ 0.64%) 72.23s 73.30s p=0.045 n=6
Emit Time 21.32s (± 9.87%) 22.05s (± 9.17%) ~ 19.38s 23.53s p=0.378 n=6
Total Time 111.04s (± 1.84%) 112.26s (± 1.89%) ~ 109.28s 114.01s p=0.093 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 265,853 265,853 ~ ~ ~ p=1.000 n=6
Types 108,438 108,438 ~ ~ ~ p=1.000 n=6
Memory used 410,402k (± 0.03%) 410,367k (± 0.02%) ~ 410,285k 410,429k p=0.810 n=6
Parse Time 4.90s (± 0.84%) 4.89s (± 0.98%) ~ 4.80s 4.93s p=0.746 n=6
Bind Time 2.07s (± 0.83%) 2.06s (± 0.51%) ~ 2.05s 2.08s p=0.566 n=6
Check Time 21.15s (± 0.22%) 21.18s (± 0.41%) ~ 21.01s 21.26s p=0.261 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 28.13s (± 0.29%) 28.14s (± 0.27%) ~ 28.02s 28.21s p=1.000 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 523,981 523,981 ~ ~ ~ p=1.000 n=6
Types 178,708 178,708 ~ ~ ~ p=1.000 n=6
Memory used 461,250k (± 0.02%) 461,296k (± 0.02%) ~ 461,212k 461,461k p=0.689 n=6
Parse Time 3.99s (± 0.47%) 4.02s (± 0.84%) ~ 3.99s 4.08s p=0.073 n=6
Bind Time 1.46s (± 1.54%) 1.47s (± 0.67%) ~ 1.46s 1.48s p=0.324 n=6
Check Time 22.52s (± 0.42%) 22.52s (± 0.56%) ~ 22.31s 22.66s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 27.98s (± 0.35%) 28.02s (± 0.40%) ~ 27.85s 28.17s p=0.689 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

gabritto commented 2 weeks ago

@typescript-bot perf test this

typescript-bot commented 2 weeks ago

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results
typescript-bot commented 2 weeks ago

@gabritto The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,154 62,154 ~ ~ ~ p=1.000 n=6
Types 50,273 50,273 ~ ~ ~ p=1.000 n=6
Memory used 192,709k (± 0.73%) 192,785k (± 0.77%) ~ 192,065k 195,786k p=0.936 n=6
Parse Time 1.65s (± 1.52%) 1.63s (± 1.58%) ~ 1.61s 1.67s p=0.514 n=6
Bind Time 0.87s (± 0.94%) 0.86s (± 0.94%) ~ 0.85s 0.87s p=0.666 n=6
Check Time 11.34s (± 0.35%) 11.33s (± 0.42%) ~ 11.27s 11.39s p=0.688 n=6
Emit Time 3.15s (± 0.80%) 3.14s (± 0.56%) ~ 3.11s 3.16s p=0.517 n=6
Total Time 17.00s (± 0.27%) 16.96s (± 0.17%) ~ 16.93s 17.00s p=0.170 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 945,172 945,172 ~ ~ ~ p=1.000 n=6
Types 408,068 408,068 ~ ~ ~ p=1.000 n=6
Memory used 1,221,963k (± 0.01%) 1,221,999k (± 0.00%) ~ 1,221,950k 1,222,087k p=0.471 n=6
Parse Time 8.21s (± 0.37%) 8.22s (± 0.33%) ~ 8.18s 8.25s p=0.375 n=6
Bind Time 2.23s (± 0.57%) 2.23s (± 0.49%) ~ 2.22s 2.25s p=0.869 n=6
Check Time 36.66s (± 0.28%) 36.68s (± 0.32%) ~ 36.58s 36.90s p=0.630 n=6
Emit Time 17.47s (± 0.45%) 17.45s (± 0.35%) ~ 17.37s 17.52s p=0.575 n=6
Total Time 64.57s (± 0.18%) 64.58s (± 0.19%) ~ 64.49s 64.76s p=0.521 n=6
mui-docs - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 1,954,602 1,954,602 ~ ~ ~ p=1.000 n=6
Types 676,397 676,397 ~ ~ ~ p=1.000 n=6
Memory used 1,753,359k (± 0.00%) 1,753,398k (± 0.00%) ~ 1,753,322k 1,753,443k p=0.128 n=6
Parse Time 9.99s (± 0.48%) 9.98s (± 0.46%) ~ 9.92s 10.06s p=0.872 n=6
Bind Time 3.36s (± 0.70%) 3.37s (± 0.83%) ~ 3.34s 3.41s p=0.935 n=6
Check Time 82.24s (± 0.41%) 82.15s (± 0.41%) ~ 81.65s 82.54s p=0.810 n=6
Emit Time 0.20s (± 2.54%) 0.21s (± 6.37%) ~ 0.19s 0.22s p=0.546 n=6
Total Time 95.79s (± 0.39%) 95.71s (± 0.39%) ~ 95.10s 96.08s p=0.810 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,215,071 1,215,421 +350 (+ 0.03%) ~ ~ p=0.001 n=6
Types 257,359 257,437 +78 (+ 0.03%) ~ ~ p=0.001 n=6
Memory used 2,323,779k (± 0.05%) 2,325,334k (± 0.02%) +1,556k (+ 0.07%) 2,324,591k 2,325,767k p=0.031 n=6
Parse Time 6.16s (± 0.79%) 6.13s (± 1.28%) ~ 6.08s 6.28s p=0.470 n=6
Bind Time 2.22s (± 1.25%) 2.24s (± 0.40%) ~ 2.23s 2.25s p=0.172 n=6
Check Time 39.97s (± 0.43%) 39.90s (± 0.18%) ~ 39.78s 39.99s p=0.689 n=6
Emit Time 3.23s (± 4.20%) 3.08s (± 0.99%) 🟩-0.15s (- 4.74%) 3.04s 3.12s p=0.013 n=6
Total Time 51.59s (± 0.58%) 51.37s (± 0.22%) ~ 51.22s 51.56s p=0.173 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,215,071 1,215,421 +350 (+ 0.03%) ~ ~ p=0.001 n=6
Types 257,359 257,437 +78 (+ 0.03%) ~ ~ p=0.001 n=6
Memory used 2,398,198k (± 0.04%) 2,399,227k (± 0.03%) ~ 2,398,272k 2,399,905k p=0.128 n=6
Parse Time 7.84s (± 0.92%) 7.85s (± 0.77%) ~ 7.75s 7.91s p=0.748 n=6
Bind Time 2.46s (± 1.52%) 2.46s (± 0.49%) ~ 2.45s 2.48s p=0.747 n=6
Check Time 50.09s (± 0.36%) 50.25s (± 0.47%) ~ 50.02s 50.68s p=0.471 n=6
Emit Time 4.01s (± 4.10%) 3.96s (± 1.38%) ~ 3.89s 4.03s p=0.521 n=6
Total Time 64.42s (± 0.33%) 64.54s (± 0.41%) ~ 64.27s 65.00s p=0.471 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 256,220 256,295 +75 (+ 0.03%) ~ ~ p=0.001 n=6
Types 103,640 103,661 +21 (+ 0.02%) ~ ~ p=0.001 n=6
Memory used 424,088k (± 0.01%) 424,291k (± 0.01%) +203k (+ 0.05%) 424,253k 424,341k p=0.005 n=6
Parse Time 4.31s (± 0.60%) 4.32s (± 0.45%) ~ 4.29s 4.34s p=0.518 n=6
Bind Time 1.60s (± 1.46%) 1.59s (± 0.32%) ~ 1.59s 1.60s p=0.276 n=6
Check Time 22.45s (± 0.45%) 22.52s (± 0.29%) ~ 22.44s 22.61s p=0.148 n=6
Emit Time 1.74s (± 1.65%) 1.72s (± 0.97%) ~ 1.70s 1.75s p=0.192 n=6
Total Time 30.11s (± 0.44%) 30.16s (± 0.19%) ~ 30.10s 30.25s p=0.572 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,824 224,824 ~ ~ ~ p=1.000 n=6
Types 93,390 93,390 ~ ~ ~ p=1.000 n=6
Memory used 369,276k (± 0.01%) 369,385k (± 0.05%) ~ 369,251k 369,730k p=0.173 n=6
Parse Time 3.68s (± 0.83%) 3.68s (± 0.68%) ~ 3.64s 3.70s p=0.807 n=6
Bind Time 1.93s (± 1.01%) 1.93s (± 1.00%) ~ 1.91s 1.96s p=0.872 n=6
Check Time 19.47s (± 0.38%) 19.44s (± 0.24%) ~ 19.40s 19.51s p=0.460 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 25.07s (± 0.32%) 25.05s (± 0.19%) ~ 24.99s 25.13s p=0.746 n=6
vscode - node (v18.15.0, x64)
Errors 4 4 ~ ~ ~ p=1.000 n=6
Symbols 2,795,894 2,795,894 ~ ~ ~ p=1.000 n=6
Types 949,958 949,958 ~ ~ ~ p=1.000 n=6
Memory used 2,924,684k (± 0.00%) 2,924,772k (± 0.00%) ~ 2,924,689k 2,924,848k p=0.092 n=6
Parse Time 16.65s (± 0.42%) 16.58s (± 0.22%) ~ 16.53s 16.63s p=0.077 n=6
Bind Time 4.97s (± 0.55%) 4.97s (± 0.32%) ~ 4.94s 4.98s p=0.742 n=6
Check Time 88.66s (± 0.60%) 88.48s (± 0.18%) ~ 88.20s 88.65s p=0.936 n=6
Emit Time 24.75s (± 7.62%) 23.91s (± 0.93%) ~ 23.62s 24.20s p=0.575 n=6
Total Time 135.03s (± 1.38%) 133.94s (± 0.27%) ~ 133.44s 134.35s p=0.298 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 265,853 265,853 ~ ~ ~ p=1.000 n=6
Types 108,438 108,438 ~ ~ ~ p=1.000 n=6
Memory used 410,405k (± 0.03%) 410,345k (± 0.01%) ~ 410,303k 410,410k p=0.378 n=6
Parse Time 3.25s (± 0.96%) 3.25s (± 0.79%) ~ 3.21s 3.27s p=0.624 n=6
Bind Time 1.41s (± 0.98%) 1.41s (± 0.98%) ~ 1.38s 1.42s p=1.000 n=6
Check Time 14.46s (± 0.24%) 14.46s (± 0.26%) ~ 14.41s 14.51s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.11s (± 0.31%) 19.11s (± 0.11%) ~ 19.09s 19.14s p=0.936 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 523,981 523,981 ~ ~ ~ p=1.000 n=6
Types 178,708 178,708 ~ ~ ~ p=1.000 n=6
Memory used 461,214k (± 0.02%) 461,228k (± 0.01%) ~ 461,178k 461,306k p=0.521 n=6
Parse Time 3.25s (± 0.64%) 3.23s (± 0.57%) ~ 3.22s 3.26s p=0.182 n=6
Bind Time 1.18s (± 0.46%) 1.18s (± 0.76%) ~ 1.17s 1.19s p=0.341 n=6
Check Time 18.16s (± 0.29%) 18.19s (± 0.31%) ~ 18.12s 18.27s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.59s (± 0.26%) 22.60s (± 0.28%) ~ 22.51s 22.67s p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,364ms (± 0.37%) 2,347ms (± 0.48%) -16ms (- 0.69%) 2,334ms 2,368ms p=0.031 n=6
Req 2 - geterr 5,102ms (± 0.53%) 5,146ms (± 0.74%) +44ms (+ 0.87%) 5,101ms 5,193ms p=0.031 n=6
Req 3 - references 288ms (± 1.40%) 289ms (± 1.34%) ~ 286ms 296ms p=0.462 n=6
Req 4 - navto 227ms (± 0.72%) 226ms (± 0.56%) ~ 225ms 228ms p=0.183 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 80ms (± 8.28%) 80ms (± 8.54%) ~ 76ms 93ms p=0.867 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,483ms (± 0.79%) 2,470ms (± 1.18%) ~ 2,417ms 2,501ms p=0.470 n=6
Req 2 - geterr 3,833ms (± 0.29%) 3,834ms (± 0.17%) ~ 3,828ms 3,846ms p=0.748 n=6
Req 3 - references 301ms (± 0.25%) 299ms (± 0.46%) -3ms (- 0.89%) 297ms 301ms p=0.014 n=6
Req 4 - navto 228ms (± 0.24%) 227ms (± 0.33%) ~ 226ms 228ms p=0.476 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 75ms (± 7.01%) 74ms (± 6.45%) ~ 71ms 84ms p=0.742 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,182ms (± 0.22%) 5,184ms (± 0.19%) ~ 5,168ms 5,199ms p=0.575 n=6
Req 2 - geterr 1,129ms (± 0.58%) 1,130ms (± 0.76%) ~ 1,125ms 1,147ms p=1.000 n=6
Req 3 - references 85ms (± 1.21%) 85ms (± 1.22%) ~ 84ms 86ms p=0.311 n=6
Req 4 - navto 447ms (± 0.26%) 449ms (± 0.76%) ~ 446ms 454ms p=0.184 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 839ms (± 0.53%) 841ms (± 1.55%) ~ 833ms 867ms p=0.809 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 156.93ms (± 0.16%) 157.02ms (± 0.18%) +0.09ms (+ 0.06%) 155.86ms 160.96ms p=0.007 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 239.28ms (± 0.14%) 239.35ms (± 0.14%) +0.07ms (+ 0.03%) 238.20ms 242.24ms p=0.045 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 235.09ms (± 0.12%) 235.35ms (± 0.15%) +0.26ms (+ 0.11%) 233.74ms 240.83ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 235.42ms (± 0.80%) 235.29ms (± 0.14%) -0.13ms (- 0.05%) 233.67ms 240.98ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

typescript-bot commented 2 weeks ago

@gabritto Here are the results of running the top 400 repos comparing main and refs/pull/58318/merge:

Everything looks good!

weswigham commented 2 weeks ago

@typescript-bot perf test this faster

Juuuust in case

typescript-bot commented 2 weeks ago

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results
typescript-bot commented 2 weeks ago

@weswigham The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,154 62,154 ~ ~ ~ p=1.000 n=6
Types 50,273 50,273 ~ ~ ~ p=1.000 n=6
Memory used 192,439k (± 0.78%) 193,658k (± 1.03%) ~ 191,803k 195,613k p=0.230 n=6
Parse Time 1.35s (± 0.77%) 1.35s (± 1.09%) ~ 1.33s 1.36s p=0.676 n=6
Bind Time 0.72s 0.72s ~ ~ ~ p=1.000 n=6
Check Time 9.57s (± 0.24%) 9.60s (± 0.42%) ~ 9.54s 9.65s p=0.291 n=6
Emit Time 2.63s (± 0.31%) 2.63s (± 0.52%) ~ 2.61s 2.65s p=0.605 n=6
Total Time 14.26s (± 0.14%) 14.29s (± 0.25%) ~ 14.24s 14.33s p=0.326 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 945,073 945,073 ~ ~ ~ p=1.000 n=6
Types 408,052 408,052 ~ ~ ~ p=1.000 n=6
Memory used 1,221,332k (± 0.00%) 1,221,323k (± 0.00%) ~ 1,221,287k 1,221,396k p=0.810 n=6
Parse Time 8.20s (± 0.53%) 8.17s (± 0.44%) ~ 8.12s 8.22s p=0.260 n=6
Bind Time 2.22s (± 1.04%) 2.21s (± 0.68%) ~ 2.20s 2.24s p=1.000 n=6
Check Time 36.33s (± 0.33%) 36.30s (± 0.40%) ~ 36.06s 36.44s p=0.936 n=6
Emit Time 17.23s (± 0.43%) 17.29s (± 0.97%) ~ 17.05s 17.57s p=0.378 n=6
Total Time 63.99s (± 0.26%) 63.97s (± 0.38%) ~ 63.64s 64.27s p=1.000 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,954,535 1,954,535 ~ ~ ~ p=1.000 n=6
Types 676,397 676,397 ~ ~ ~ p=1.000 n=6
Memory used 1,752,887k (± 0.00%) 1,752,949k (± 0.00%) ~ 1,752,891k 1,753,021k p=0.078 n=6
Parse Time 8.06s (± 0.93%) 8.03s (± 0.51%) ~ 7.98s 8.09s p=0.520 n=6
Bind Time 2.72s (± 0.43%) 2.72s (± 0.19%) ~ 2.72s 2.73s p=0.542 n=6
Check Time 67.18s (± 0.24%) 67.18s (± 0.27%) ~ 67.02s 67.51s p=0.810 n=6
Emit Time 0.16s 0.16s (± 2.52%) ~ 0.16s 0.17s p=0.405 n=6
Total Time 78.12s (± 0.16%) 78.10s (± 0.21%) ~ 77.92s 78.37s p=0.467 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,208,915 1,209,024 +109 (+ 0.01%) ~ ~ p=0.001 n=6
Types 255,383 255,395 +12 (+ 0.00%) ~ ~ p=0.001 n=6
Memory used 2,307,557k (± 0.04%) 2,307,399k (± 0.02%) ~ 2,306,588k 2,308,227k p=0.810 n=6
Parse Time 6.07s (± 0.50%) 6.06s (± 0.61%) ~ 6.00s 6.11s p=0.688 n=6
Bind Time 2.23s (± 0.90%) 2.23s (± 0.72%) ~ 2.21s 2.25s p=1.000 n=6
Check Time 39.82s (± 0.46%) 39.82s (± 0.35%) ~ 39.65s 40.06s p=0.936 n=6
Emit Time 3.14s (± 2.75%) 3.16s (± 4.42%) ~ 3.04s 3.41s p=0.936 n=6
Total Time 51.28s (± 0.34%) 51.29s (± 0.53%) ~ 50.99s 51.63s p=1.000 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,208,915 1,209,024 +109 (+ 0.01%) ~ ~ p=0.001 n=6
Types 255,383 255,395 +12 (+ 0.00%) ~ ~ p=0.001 n=6
Memory used 2,382,115k (± 0.04%) 2,382,713k (± 0.04%) ~ 2,381,083k 2,384,129k p=0.378 n=6
Parse Time 7.67s (± 0.73%) 7.67s (± 0.53%) ~ 7.62s 7.74s p=0.936 n=6
Bind Time 2.50s (± 0.93%) 2.55s (± 3.81%) ~ 2.51s 2.75s p=0.462 n=6
Check Time 49.68s (± 0.28%) 49.64s (± 0.44%) ~ 49.24s 49.90s p=0.810 n=6
Emit Time 3.90s (± 2.33%) 3.89s (± 1.66%) ~ 3.81s 3.98s p=0.689 n=6
Total Time 63.77s (± 0.30%) 63.76s (± 0.29%) ~ 63.58s 64.10s p=0.936 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 253,659 253,639 -20 (- 0.01%) ~ ~ p=0.001 n=6
Types 101,551 101,548 -3 (- 0.00%) ~ ~ p=0.001 n=6
Memory used 419,363k (± 0.01%) 419,398k (± 0.01%) ~ 419,373k 419,441k p=0.199 n=6
Parse Time 4.20s (± 0.83%) 4.21s (± 0.49%) ~ 4.19s 4.24s p=0.622 n=6
Bind Time 1.60s (± 1.62%) 1.61s (± 1.57%) ~ 1.56s 1.63s p=0.256 n=6
Check Time 22.36s (± 0.40%) 22.37s (± 0.45%) ~ 22.23s 22.48s p=1.000 n=6
Emit Time 1.68s (± 0.97%) 1.68s (± 1.53%) ~ 1.64s 1.71s p=0.806 n=6
Total Time 29.84s (± 0.34%) 29.87s (± 0.47%) ~ 29.67s 30.03s p=0.689 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 34 34 ~ ~ ~ p=1.000 n=6
Symbols 224,824 224,824 ~ ~ ~ p=1.000 n=6
Types 93,390 93,390 ~ ~ ~ p=1.000 n=6
Memory used 368,977k (± 0.01%) 368,983k (± 0.00%) ~ 368,956k 369,005k p=1.000 n=6
Parse Time 3.64s (± 0.56%) 3.65s (± 0.77%) ~ 3.64s 3.71s p=0.246 n=6
Bind Time 1.93s (± 1.76%) 1.90s (± 0.79%) ~ 1.89s 1.93s p=0.170 n=6
Check Time 19.42s (± 0.50%) 19.41s (± 0.34%) ~ 19.34s 19.50s p=0.809 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 24.98s (± 0.44%) 24.97s (± 0.29%) ~ 24.87s 25.05s p=0.936 n=6
vscode - node (v18.15.0, x64)
Errors 4 4 ~ ~ ~ p=1.000 n=6
Symbols 2,796,289 2,796,289 ~ ~ ~ p=1.000 n=6
Types 949,823 949,823 ~ ~ ~ p=1.000 n=6
Memory used 2,924,219k (± 0.00%) 2,924,357k (± 0.00%) +137k (+ 0.00%) 2,924,319k 2,924,386k p=0.020 n=6
Parse Time 16.53s (± 0.45%) 16.54s (± 0.22%) ~ 16.48s 16.59s p=0.630 n=6
Bind Time 4.92s (± 0.62%) 4.92s (± 0.46%) ~ 4.90s 4.95s p=1.000 n=6
Check Time 88.09s (± 0.39%) 87.84s (± 0.35%) ~ 87.47s 88.13s p=0.470 n=6
Emit Time 26.06s (± 9.63%) 23.70s (± 0.46%) ~ 23.56s 23.84s p=0.065 n=6
Total Time 135.60s (± 1.91%) 133.00s (± 0.26%) -2.60s (- 1.92%) 132.63s 133.45s p=0.045 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 265,779 265,779 ~ ~ ~ p=1.000 n=6
Types 108,400 108,400 ~ ~ ~ p=1.000 n=6
Memory used 409,822k (± 0.02%) 409,888k (± 0.03%) ~ 409,765k 410,044k p=0.199 n=6
Parse Time 4.86s (± 0.60%) 4.86s (± 0.58%) ~ 4.83s 4.89s p=0.685 n=6
Bind Time 2.08s (± 1.37%) 2.06s (± 0.59%) ~ 2.05s 2.08s p=0.287 n=6
Check Time 21.03s (± 0.48%) 21.03s (± 0.28%) ~ 20.95s 21.09s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 27.98s (± 0.33%) 27.95s (± 0.26%) ~ 27.86s 28.05s p=0.748 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 523,904 523,904 ~ ~ ~ p=1.000 n=6
Types 178,697 178,697 ~ ~ ~ p=1.000 n=6
Memory used 460,732k (± 0.01%) 460,802k (± 0.01%) +70k (+ 0.02%) 460,767k 460,838k p=0.006 n=6
Parse Time 2.68s (± 0.30%) 2.70s (± 0.59%) ~ 2.68s 2.72s p=0.118 n=6
Bind Time 0.98s (± 0.77%) 0.98s (± 0.56%) ~ 0.98s 0.99s p=0.476 n=6
Check Time 15.40s (± 0.23%) 15.42s (± 0.35%) ~ 15.36s 15.49s p=0.936 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.07s (± 0.24%) 19.10s (± 0.28%) ~ 19.05s 19.17s p=0.228 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

weswigham commented 2 weeks ago

Minus 2% time in vscode? Neat. They must have had quite a few duplicates.

jakebailey commented 2 weeks ago

That result is right on the cusp of significance, and its before value is +/- 2%, so it's possible that it's not a real result, unfortunately. We're still p-hacking ourselves...

gabritto commented 2 weeks ago

That result is right on the cusp of significance, and its before value is +/- 2%, so it's possible that it's not a real result, unfortunately. We're still p-hacking ourselves...

Yep, I'd suspect it's just noise. I don't expect anything to get faster, and I think nothing changed on extended tests, so doesn't seem like vscode has duplicate diagnostics. On the upside, nothing got significantly worse either :)

weswigham commented 2 weeks ago

We're still p-hacking ourselves...

Should we un-fix n and instead fix p?

jakebailey commented 2 weeks ago

Running the vscode benchmark already takes half an hour, I don't see us being able to change n that much.