Open whitespine opened 1 year ago
FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in https://github.com/microsoft/TypeScript/issues/50515.
FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.
I will attempt this as soon as i have recruited someone with enough ram to run it without crashing :)
4.8.4 Diagnostic information:
Files: 1023
Lines of Library: 28146
Lines of Definitions: 218302
Lines of TypeScript: 25936
Lines of JavaScript: 0
Lines of JSON: 0
Lines of Other: 0
Nodes of Library: 118224
Nodes of Definitions: 564026
Nodes of TypeScript: 115815
Nodes of JavaScript: 0
Nodes of JSON: 0
Nodes of Other: 0
Identifiers: 301992
Symbols: 865373
Types: 208799
Instantiations: 365698
Memory used: 909729K
Assignability cache size: 27600
Identity cache size: 4422
Subtype cache size: 7237
Strict subtype cache size: 9428
Tracing time: 0.17s
I/O Read time: 0.06s
Parse time: 0.96s
ResolveModule time: 0.08s
ResolveTypeReference time: 0.00s
Program time: 1.27s
Bind time: 0.62s
Check time: 2.85s
printTime time: 0.00s
Emit time: 0.00s
Dump types time: 9.03s
Total time: 4.74s
5.1.0-dev.20230305 Diagnostic information:
Files: 1029
Lines of Library: 29397
Lines of Definitions: 218302
Lines of TypeScript: 25916
Lines of JavaScript: 0
Lines of JSON: 0
Lines of Other: 0
Identifiers: 302941
Symbols: 6242486
Types: 702542
Instantiations: 417693
Memory used: 5496499K
Assignability cache size: 31947
Identity cache size: 4714
Subtype cache size: 5307
Strict subtype cache size: 7579
Tracing time: 0.08s
I/O Read time: 0.04s
Parse time: 0.94s
ResolveModule time: 0.09s
ResolveTypeReference time: 0.00s
Program time: 1.19s
Bind time: 0.49s
Check time: 57.68s
printTime time: 0.00s
Emit time: 0.00s
Dump types time: 7.27s
Total time: 59.37s
Notably there is an about 5x increase in memory usage, as well as a >5x increase in the "Types" and "Symbols".
Now that I can actually build on v10 there several new errors, most interesting among them the following:
src/module/item/license-sheet.ts:37:23 - error TS2590: Expression produces a union type that is too complex to represent.
The line is ```TS for (let d of docs as LancerItem[])
// Find the assoc frame
for (let et of [EntryType.FRAME, EntryType.MECH_SYSTEM, EntryType.MECH_WEAPON, EntryType.WEAPON_MOD]) {
let pack = game.packs.get(`world.${et}`);
if (pack) {
let docs = await pack.getDocuments({ "system.license": license.system.key });
for (let d of docs as LancerItem[]) {
let rank = (d as any).system.license_level as number;
while (unlocks.length <= rank) {
unlocks.push([]);
}
unlocks[rank].push(d);
}
}
}
Where docs
is of type:
docs: StoredDocument<LancerActor | Scene | LancerItem | JournalEntry | Macro | Playlist | RollTable>[]
.
Here's where "it's a big project" sort of bites us, in that I can't really minimally represent these types here in this comment. Each is a unique class, all with a shared parent class.
Maybe this is indicative of large union types being the root of our problem?
...
Observing the trace, the file src/module/comp-builder.ts
is responsible for an entire 53 SECONDS of the total 57 second Check Time. It produces no errors.
Unsure how to proceed from here.
i had a quite similar issue recently after upgrading typescript. i can't remember the details, but it seems the problem was that under some circumstances tsc started creating a lot of types internally when asked to infer the type of a (typewise) complicated function. in the end, writing const foo:FunctionType = (it) => ...
instead of const foo = (it:FunctionArgs):FunctionReturn => ...
solved the issue. maybe something similar is happening here?
Ran into this issue myself today on a large project.
Similar regression on 5.0.2. For me, 4.8.3 is ok.
thanks for the exceptional bug report @whitespine!
I guess there's another issue about JavaScript heap out of memory
errors over here (about 4.3
):
This issue specifically mentions a OOM when going from 4.8 to 4.9; if you are experiencing an OOM that isn't explicitly when you go from 4.8 to 4.9, your issue is most certainly different and should be reported as a new issue with its own repro.
I was able to solve the OOM issue for myself with
NODE_OPTIONS=--max-old-space-size=8192 tsc
Since @Raynos solution is really just a stop gap. What is needed to further this ticket. We also have a large project that works fine on 4.8.4
but cant upgrade to 4.9 or 5.0 because tsc gets an OOM error
Here is what I have gathered from running some test.
Test Ran with
NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --diagnostics --extendedDiagnostics --incremental false
Here is 4.8
Files: 2979
Lines of Library: 33311
Lines of Definitions: 233669
Lines of TypeScript: 59133
Lines of JavaScript: 0
Lines of JSON: 943
Lines of Other: 0
Nodes of Library: 148246
Nodes of Definitions: 723896
Nodes of TypeScript: 250983
Nodes of JavaScript: 0
Nodes of JSON: 2256
Nodes of Other: 0
Identifiers: 382980
Symbols: 597111
Types: 216450
Instantiations: 7252666
Memory used: 684086K
Assignability cache size: 202376
Identity cache size: 3213
Subtype cache size: 12303
Strict subtype cache size: 4719
I/O Read time: 0.11s
Parse time: 1.95s
ResolveModule time: 0.58s
ResolveTypeReference time: 0.01s
Program time: 3.04s
Bind time: 1.00s
Check time: 16.47s
printTime time: 0.00s
Emit time: 0.00s
Total time: 20.50s
Here is 4.9
Files: 2980
Lines of Library: 33853
Lines of Definitions: 233669
Lines of TypeScript: 59133
Lines of JavaScript: 0
Lines of JSON: 943
Lines of Other: 0
Identifiers: 384065
Symbols: 571900
Types: 18112640
Instantiations: 7055141
Memory used: 5726426K
Assignability cache size: 10914208
Identity cache size: 3215
Subtype cache size: 11958
Strict subtype cache size: 4719
I/O Read time: 0.15s
Parse time: 1.72s
ResolveModule time: 0.47s
ResolveTypeReference time: 0.01s
Program time: 2.65s
Bind time: 0.79s
Check time: 199.89s
printTime time: 0.00s
Emit time: 0.00s
Total time: 203.33s
I also generated a trace
From looking into the trace. It seems to be stuck in a loop related to the i18next
& react-i18next
This is basically the part that loops (if I followed the trace right)
import type { Namespace, TFuncKey } from 'i18next';
import { i18n } from './i18n';
export interface I18nText<N extends Namespace = Namespace> {
namespace: N;
keyName: TFuncKey<N>; // <-- TFuncKey seems to be the culprit
}
That lib is for sure complicated typewise. But it worked as it should in 4.8 and breaks compilation in 4.9 and 5.0
Anything else I can provide? @jakebailey
We've made some improvements recently that feel like this issue; can you try the nightly build?
Using Version "5.1.0-dev.20230328"
Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental false
Result
/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:114282
throw e;
^
RangeError: Value undefined out of range for undefined options property undefined
at Map.set (<anonymous>)
at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60159:24)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)
at eachTypeRelatedToType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59946:25)
at unionOrIntersectionRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59768:174)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:39)
at structuredTypeRelatedToWorker (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60559:25)
at structuredTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60172:21)
at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60142:19)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)
Note
This took about 4mins to run before failing
Uh, hm, well, it's certainly not supposed to crash anymore. If you have the time, if you can roll it back by date to find when it started crashing, that would be helpful.
Phew! Ok finally I got it.
Last version that did not crash. 5.0.0-dev.20221103
Which makes the first version that crashed: 5.0.0-dev.20221108
Note that they were still very slow. And still needed NODE_OPTIONS=--max-old-space-size=8192` set.
Small Note
From
5.0.0-dev.20230210
and lower the error message slightly changed from what I posted earlier to
/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:113758
throw e;
^
RangeError: Value undefined out of range for undefined options property undefined
at Map.set (<anonymous>)
at getIndexedAccessTypeOrUndefined (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56795:28)
at getIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56764:12)
at /home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:68
at map (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:209:19)
at distributeIndexOverObjectType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:21)
at getSimplifiedIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56710:37)
at getSimplifiedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56683:55)
at getNormalizedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:58949:481)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59262:23)
Oh, yikes, those are the versions before and after the module transform (where things were temporarily broken). Is your project somewhere that I can test it?
It is not. Tomorrow I can perhaps try and create a repo that replicates some of the i18next stuff we are doing with types and see if I can get it to have the same issue.
@jakebailey Well hey it wasn't that bad, I was able to replicate it with little code
https://github.com/JarekToro/TSC-Regression-test-5.0
I understand this to be the offending line. https://github.com/JarekToro/TSC-Regression-test-5.0/blob/main/utils.ts#L15
If you remove that line. Or remove the function in its entirety. It builds correctly.
Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental falseΒ
To Test
I'm trying to look at this, but it seems like this thread contains many different OOMs and claims about which versions do and don't work.
The only repro I have is @JarekToro's, but that explicitly says that something changed in 5.0. But, if I go back to TS 4.8, that code fails to compile entirely, saying that it's excessively deep. 4.9 hangs, as does 5.0. Is it supposed to complete?
But, that's still not the original reported issue. I guess what I should be checking is https://github.com/Eranziel/foundryvtt-lancer/tree/oom?
Testing the case from the original issue, a bisect points to #51151. @weswigham
Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.
My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire aws-sdk
v2 library loaded into my project and each copy of aws-sdk
has an insanely large amount of symbols.
I don't have a reproduction example at hand, but npm dedupe aws-sdk
also resolved my OOM issue to reduce the amount of symbols.
Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.
@jakebailey That's good to hear, So yes in regards to my reproduction I was trying to focus on what version the build failed at. Which while related is technically different. Then the original issue. And while I simplified the code base to reproduce the crash. I didn't check to see if it passed on 4.8.4. If I have time I will try to make a separate repro of the slow down difference.
My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire
aws-sdk
v2 library loaded into my project and each copy ofaws-sdk
has an insanely large amount of symbols.I don't have a reproduction example at hand, but
npm dedupe aws-sdk
also resolved my OOM issue to reduce the amount of symbols.
Yeah, that might just be a different issue. We've had similar problems on DefinitelyTyped regarding that. If you can see something change between two different TS versions in the --extendedDiagnostics
, that might be interesting, though.
Have same issue with 5.0.4
along with 5.1.0-beta
also using i18next
along with react-i18next
Have same issue with
5.0.4
along with5.1.0-beta
also usingi18next
along withreact-i18next
Same issue for me, and we are also using i18next
and react-i18next
.
Facing the same issue with i18next and react-i18next, tested with typescript@5.1.0-dev.20230501 and still no luck π Any timeline on a fix for this issue?
I have a workaround working for ts@4.9.5. I forked off of @JarekToro's repo and implemented it:
https://github.com/wrdecurtins/TSC-4.9.5-workaround
This comment on an i18next issue suggested removing resources
from CustomTypeOptions
to improve performance during tsc compilation, which seems to work well for ts@4.9.5:
https://github.com/i18next/react-i18next/issues/1600#issuecomment-1374277198
You can also redeclare the TFunction
interface to make it less complicated.
declare module 'i18next' {
interface TFunction {
(key: string, options?: {[key: string]: string}): string
}
}
These changes reduce the type safety of the i18next libraries, so I wouldn't recommend using this as anything other than a temporary workaround to get to ts@4.9.5.
Unfortunately this doesn't seem to do anything to fix the issue for ts@5+.
There is a pretty good vscode i18next extension that can help restore type safety and autofill i18n keys maintained by the lokalise team:
Also rolling back the i18next package to v21.10.0 and react-i18next to v11.16.9 worked for me.
facing the same problem when upgrading to TS 5+
our tsc build slowly grows its memory usage until it eventually crashes with @whitespine 's error
100% regression issue... 5.0.4 used to work fine after upgrading to the current latest version 5.1.3 all hell break loose! Very slow builds and eventually crashes. Seems like some kind of endless loop in the compiler.
If the 5.0 works and 5.1.3 doesn't no way its a 'classic' out of memory issue, this is 100% an "endless loop" or "circular dependencies" issue within tsc
.
Seems like tsc can't handle interface definition files anymore, in this case from lodash
?
node_modules/@types/lodash/common/common.d.ts:194:15 - error TS2589: Type instantiation is excessively deep and possibly infinite.
194 interface Object<T> extends LoDashImplicitWrapper<T> {
~~~~~~
node_modules/@types/lodash/common/common.d.ts:206:15 - error TS2430: Interface 'ObjectChain<T>' incorrectly extends interface 'LoDashExplicitWrapper<T>'.
The types returned by 'entries().pop()' are incompatible between these types.
Type 'CollectionChain<string | T[keyof T]>' is missing the following properties from type 'ObjectChain<[string, any]>': assign, assignIn, assignInWith, assignWith, and 11 more.
206 interface ObjectChain<T> extends LoDashExplicitWrapper<T> {
~~~~~~~~~~~
Related? https://github.com/microsoft/TypeScript/issues/34933
Anyway, adding "skipLibCheck": true
to the tsconfig.json did mitigate the issue for now.
This issue explicitly says something broke between 4.8 and 4.9. If you have a problem going from 5.0 to 5.1, it is almost assuredly another issue and I would suggest filing a new one or looking for another recent issue about your problem which mentions 5.1.
Updated from 4.9.5
to 5.1.3
today and having the same exact issue. Build hangs until it runs OOM. Using references
and paths
in a Yarn PnP monorepo.
@ernestostifano As above, this issue is intended to be for a specific regression between 4.8 and 4.9; if you're jumping from 4.9 to 5.1, it's unlikely to be the same thing. Can you please file a new issue and also test 5.0 to better narrow the range?
@jakebailey maybe, if more users report this issue as well. Despite different ts versions the developers actually might look into the problem. And solve the issue for everyone.
Keep in mind that priorities need to be set on these kind of huge projects, with more than 5000 issues reported on github!
We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.
We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.
You have a point. Going to watch #54517 from now on, even though the error is the same.
@jakebailey but there are several comments in this thread referring to versions above 5.0.0
. Maybe your affected range is not correct. We have a really big project too, everything was working fine on 4.9.5
and just updating TypeScript to 5.1.3
triggered this issue. The funny thing is that the build hangs on a package containing i18next
as a dependency, just like the examples above.
The build just take forever until it crashes going OOM.
We downgraded to 5.0.4
and the issue remains.
There is a comment from @JarekToro above reporting that the first affected version is 5.0.0-dev.20221108
.
Since ours is a very sensitive project, we will just have to downgrade to 4.9.5
until this issue gets solved.
We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.
@jakebailey I understand your point and me too will watch https://github.com/microsoft/TypeScript/issues/54517 instead, but it really looks like we are talking about the same issue here.
It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions.
OK, created a separate issue: https://github.com/microsoft/TypeScript/issues/54542
For everyone that are struggling with i18n
, bumping packages to following versions seems to fixed issue with OOM for Typescript v4.9.5
for me.
Previous:
"i18next": "22.5.0"
"react-i18next": "12.3.1"
New:
"i18next": "23.2.8"
"react-i18next": "13.0.2"
However, now I've been bombarded with a lot of type errors π
@ashylen unfortunately the latest versions of i18next didn't resolve the issue for me. See reproducible example here https://github.com/i18next/i18next/issues/1921#issuecomment-1614113491
Still getting this issue. I'm getting the following error:
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
when running tsc --pretty --noEmit
even if I set the space size like the following:
cross-env NODE_OPTIONS=--max-old-space-size=10240 tsc --pretty --noEmit
Typescript compiler 4.8.4
works, I downgraded to build my project
This thread has lots and lots of people with likely different issues; I've recently published a tool which makes it easy to bisect issues in TypeScript without knowing how to build TypeScript itself: https://www.npmjs.com/package/every-ts#bisecting
It'd be extremely helpful for those who know which versions do and don't work to use this tool to identify which change in particular broke things; doing so will allow us to categorize these issues.
We have the same issue as original and it is also with i18next
, it's been blocking us from upgrading the library for almost a year :/
Do you have a repo that can be cloned and tested consistently?
Hi, I didn't manage to create a repo reproducing the error but I have found a fix for the issue in i18next, I will report back there with a PR in 2-3 weeks if everything has worked as expected in the meantime, and link it here so that you can analyse wether it's something ts could get better at. Basically the issue has to do with values and inferences within conditional types being computed when they shouldn't (moving a ternary up the type stack and doing some type caching with const fixed the issue (not sure what I say strictly makes sense as I'm a noob in typing jargon but you get the idea)).
https://github.com/i18next/i18next/issues/2138#issuecomment-2036850539
In the meantime here is what seems to do the trick, in case someone wants to have a look https://github.com/i18next/i18next/compare/master...GreenGoTech:i18next:master
FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.
doing this helped me trace the actual problem. I had some recursive types in my code that were causing this issue.
Bug Report
When updating typescript within our project from version 4.7.3 to a more recent version, our project began experiencing OOM issues. Specifically, it is completely unable to build via
tsc
or eventsc --noEmit
. Ours is a somewhat large project, so I couldn't point to a minimal reproduction (sorry!). I have 16GB of RAM so I doubt its a "classic" memory error, seems more likely something started infinitely using memory during a version upgrade.π Search Terms
tsc 4.9 4.9.5 Out of Memory Heap Out of Memory Crash Killed Silent Error (Am i doing this right?)
π Version & Regression Information
Error began occuring when updating from typescript 4.7.3 to any 4.9.3^ version. Issue persists into Version 5.1.0-dev.20230303 (nightly at time of testing). Issue is NOT present in v4.8.4, or any prior version we tried (that satisfied our dependencies, anyways.)
π» Code
Branch state is: https://github.com/Eranziel/foundryvtt-lancer/tree/oom
Was unable to produce a minimally viable reproduction, as the OOM error gives next to no context as to where the issue occurs or why. Sorry. More than willing to provide more investigative information, but I don't really know what there is to provide.
π Actual behavior
It crashes on more recent versions for no clear reason.
π Expected behavior
I'd rather it didn't.