Open taoqf opened 7 years ago
@sandersn , Could this be fixed?
Two questions:
The gulp-typescript setup changes options.module=1. Does tsc still succeed when you set the equivalent module option in tsconfig? It's commonjs right now and I'm not sure that 1 means commonjs
Afterwards, if tsc still works, then it sounds like a gulp-typescript bug. The stack is in emit when it runs out of memory. This is very unusual for vanilla tsc, so I would first suspect a configuration/setup bug in gulp-typescript.
Also, @rbuckton, does that crash location mean anything to you?
1 means comonjs 2 amd 3 umd
I set options.module=1
this because i have multi tasks in gulp.
I guess gulp may use more memory than tsc because it will store files in memory steam.
I have tested if I reduce the number of the file it will succeed.
looks like the gulp typescript + typescript memory usage pushes you over the limit. I would say gulp-typescript should take the first stab at this issue.
@mhegazy , I got this Error just now:
<--- Last few GCs --->
[22042:0x3696020] 34073 ms: Mark-sweep 1403.3 (1447.3) -> 1403.3 (1442.3) MB, 877.7 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 878 ms) last resort
[22042:0x3696020] 34966 ms: Mark-sweep 1403.3 (1442.3) -> 1403.3 (1442.3) MB, 891.5 / 0.0 ms last resort
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2a16e0b9cef1 <JSObject>
1: declareSymbol [/home/taoqf/.nvm/versions/node/v8.4.0/lib/node_modules/typescript/lib/tsc.js:~18571] [pc=0x1b65f5d1bf8b](this=0x3f86308890d9 <JSGlobal Object>,symbolTable=0x10f596065381 <Map map = 0x17f790d183b9>,parent=0x220cdb20d551 <Symbol map = 0x17f790d19019>,node=0x14238561b021 <Node map = 0x17f790d14e19>,includes=4,excludes=0)
2: bindPropertyOrMethodOrAccessor [/home/taoqf/.nvm/...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x13740dc [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
7: 0x1b65f58040dd
[1] 22042 abort tsc
using tsc
.
can you share your project
+1
I have same problem. I can't share my project because our enterprise project. This is not appear in normal compile environment, but almost appear in below environment.
@taoqf @swlee60 can you try typescript@next
? The recent change #17984 improved checker resource usage for recursive types references. (eg class List<T> { map<U>(f): List<U> }
). This doesn't directly address the problem of huge projects, but it's likely to help since huge projects typically have some specialised collections somewhere.
If that doesn't help, is there some way you can share your project? I don't have any other starting point for debugging since eliminating gulp-typescript from the crash. I can sign an NDA if needed.
@sandersn
I used typescript@next
but problem still appears.
Problems occur on both sides of awesome-type-script-loader
and ts-loader
$ tsc --version Version 2.6.0-dev.20170824
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25546 ** 20 0 2763m 1.5g 16m R 99.6 39.9 3:30.55 node
webpack: Compiling...
<--- Last few GCs --->
551020 ms: Mark-sweep 1288.8 (1408.1) -> 1288.8 (1412.1) MB, 876.2 / 0.0 ms [allocation failure] [GC in old space requested]. 551884 ms: Mark-sweep 1288.8 (1412.1) -> 1288.8 (1412.1) MB, 864.3 / 0.0 ms [allocation failure] [GC in old space requested]. 552797 ms: Mark-sweep 1288.8 (1412.1) -> 1297.8 (1408.1) MB, 912.2 / 0.1 ms [last resort gc]. 553713 ms: Mark-sweep 1297.8 (1408.1) -> 1306.9 (1408.1) MB, 916.7 / 0.1 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3f6911dcfb51
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/home1/irteam/apps/bin/node]
2: 0x1098b2c [/home1/irteam/apps/bin/node]
3: v8::Utils::ReportApiFailure(char const, char const) [/home1/irteam/apps/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [/home1/irteam/apps/bin/node]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/home1/irteam/apps/bin/node]
6: v8::internal::TypeFeedbackVector::New(v8::internal::Isolate, v8::internal::Handle
@sandersn same here.
$ tsc --version
Version 2.6.0-dev.20170824
$ tsc
<--- Last few GCs --->
[26137:0x3b6b020] 37623 ms: Mark-sweep 1403.3 (1445.3) -> 1403.3 (1440.8) MB, 861.8 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 862 ms) last resort
[26137:0x3b6b020] 38487 ms: Mark-sweep 1403.3 (1440.8) -> 1403.3 (1440.8) MB, 864.1 / 0.0 ms last resort
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0xc3bfff9cef1 <JSObject>
1: declareSymbol(aka declareSymbol) [/home/taoqf/.nvm/versions/node/v8.4.0/lib/node_modules/typescript/lib/tsc.js:~14980] [pc=0xac3a15e8497](this=0xc3bfff82241 <undefined>,symbolTable=0x1eb8b183ecc9 <Map map = 0x4a1246983b9>,parent=0x29b2e1ed5e61 <Symbol map = 0x4a12469a4b9>,node=0x2cc77d79521 <Node map = 0x4a124695a79>,includes=4,excludes=0,isReplaceableByMethod=0xc3bfff82241 <undefined>)
...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x13740dc [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
7: 0xac3a14040dd
Aborted
@sandersn I'm totally ignorant of tsc internals, but I might have a thought worth chasing. If I understand it correctly, the fix you provided (PR https://github.com/Microsoft/TypeScript/pull/17984) seems incomplete, in at least a couple of ways:
extends
) types.FWIW!!! I suspect these may be subcases of broader sets of unaddressed cases. In any event, my main reason for writing is to provoke you to think along these lines for missed cases. If you're not one yourself, you might want to consult a Typescript "language lawyer".
@swlee60 Out of curiosity, does your crash disappear if you disable sourcemaps (source-map
is in your crash log, after all)?
@weswigham Do you mean webpack devtool
option? Whether I set inline-source-map
or eval
, heap overflow always comes up.
Interesting. Looks like @taoqf's project runs out of memory in the binder. Not much has changed in the binder recently, though.
@taoqf, what is the version of Typescript that you used before 2.4.1? Your project fails on 2.3 for me as well.
On master (I haven't checked with 2.3 or 2.4), the AST alone brings memory usage to 1.1 GB, which means that every byte of source results in almost 40 bytes in the parse tree. That leaves the compiler about 300 MB to bind and check with. It is unable to do so.
We need to think about architectural ways to reduce parse tree overhead to make sure that Typescript can scale to very large projects. Specifically, how much overhead we incur when parsing common patterns like object literals.
As a workaround, @taoqf, can you split your compilation into multiple compilations? Then you could compile it in multiple pieces. You can use .d.ts files, either generated or manually created, as interfaces between the split compilations.
@taoqf, what is the version of Typescript that you used before 2.4.1?
I always used next version.
<--- Last few GCs --->
64156 ms: Mark-sweep 1302.7 (1435.3) -> 1298.6 (1435.3) MB, 2648.5 / 0.0 ms [allocation failure] [GC in old space requested].
66567 ms: Mark-sweep 1298.6 (1435.3) -> 1298.6 (1435.3) MB, 2411.3 / 0.0 ms [allocation failure] [GC in old space requested].
69175 ms: Mark-sweep 1298.6 (1435.3) -> 1306.1 (1409.3) MB, 2607.0 / 0.0 ms [last resort gc].
71539 ms: Mark-sweep 1306.1 (1409.3) -> 1313.8 (1409.3) MB, 2363.7 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2dd8a203fa99 <JS Object>
1: set [native collection.js:~252] [pc=0x27ff7fb0d83b] (this=0x3c34b6f24d71 <a Map with map 0x230c6e7113b1>,r=0x1039b0b29051 <String[13]: 896958,897404>,z=0x1039b0b29079 <a TypeObject with map 0x230c6e796f31>)
2: getTypeWithThisArgument [/home/jenkins/workspace/Project/web-app/node_modules/typescript/lib/typescript.js:~31577] [pc=0x27ff7fbe98bc] ...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [@angular/cli]
2: 0x10d368c [@angular/cli]
3: v8::Utils::ReportApiFailure(char const*, char const*) [@angular/cli]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [@angular/cli]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [@angular/cli]
6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [@angular/cli]
7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [@angular/cli]
8: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashMap>) [@angular/cli]
9: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [@angular/cli]
10: 0x27ff7e2060c7
@tiger-seo can you open a new issue? There are multiple reasons for the compiler to run out of memory and your reason is likely different from @taoqf.
@taoqf if compilation worked previously, what else changed? Did your project add a lot more files? Did the individual files get a lot bigger? The project you gave me fails on 2.3, 2.4 and 2.5 and I expect it would fail on earlier versions because the AST memory footprint hasn't grown much recently.
It never worked , I compiled in multiple pieces in my project now.
@sandersn , I tried to use gulp-typescript and --max-old-space-size=4096 to run script, it will be ok, it would be very nice if tsc and gulp-typescript could accept this option.
I recently ran into the same issue using ts-loader and webpack after upgrading Typescript from 2.3.4 to 2.5.2. I downgraded to 2.4.2 and 2.4.1 and the issue persisted. With 2.4.0, however, everything works as expected.
@troeggla How big is your project? If the project is less than 10 MB of source, then it's probably a different cause.
Can you
typescript@next
?Randomly getting this on my project (using nightly, tsc in watch mode), not sure how to reproduce, it just happens sometimes. 7GB/8GB of RAM ATM, but plenty of swap and pagefile (9GBish)
07:28:49 - File change detected. Starting incremental compilation...
<--- Last few GCs --->
[1592:000002842FE68000] 7027823 ms: Mark-sweep 1401.5 (1465.4) -> 1401.5 (1465.4) MB, 4599.6 / 0.0 ms last resort
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0000033CF2C1CEA9 <JSObject>
1: set [native collection.js:~247] [pc=000000C2A8FE6697](this=000000F425FBE6F1 <Map map = 000003AA6FD983B9>,p=000003A9ED17AB21
<String[9]: @Telefone>,x=000003A9ED17AB51 <Type map = 000003AA6FDC6CF9>)
2: getLiteralType(aka getLiteralType) [C:\nodejs\node_modules\typescript\lib\tsc.js:~24883] [pc=000000C2A8D93E76](this=0000033C
F2C02241 <undefined>,value=000000CEC1A9C961 <String[8]: Telef...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
The terminal process terminated with exit code: 3
project is not too big, 36 typescript files, but only 120kb in total
@sandersn Thanks for looking into this! Yeah, my project is well below 10 MB (excluding packages though) and the resulting bundle comes to about 1.9MB (without minification). Building with typescript@next
compiles as it did with Typescript 2.4.0.
If you want, I can give you access to a private repository.
@troeggla If the nightly typescript is working for you then I think the recent bug fixes apply and we don't need your source.
@pocesar Can you open a separate issue? For only 120k of source you do not have the same issue as @taoqf.
Total RAM doesn't make a difference unless you have configured node to use more than 1.4 GB. You can try raising it to see if the problem goes away, and try lowering it to see if you can make it reproduce reliably. I think the flag is --max-old-space-size=8192
(for 8 GB).
You can also try compiling with --diagnostics
to get a count of types and symbols. More than half a million types is a suspicious number.
@sandersn Hello, When do you think this problem will be solved?
My team also has large project, and we changed and fixed TypeScript version to 2.4.0 because upper TS 2.4.1 caused this FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
issue.
We want to use awesome latest TypeScript's features, like 2.5, 2.6, ... or 3.0 and later! But we don't know when is this problem solved, so we can't upgrade TypeScript anymore.
If you can, please let me know the progress of this issue.
@cheerun Try typescript@next
. We have fixed all but a two of the known issues there, and those fixes will also ship with 2.5.3.
If you have 10s of megabytes of source then you may still have a problem; that's what this issue tracks.
The OP seems to be caused by a large object structure, @sandersn's comment in https://github.com/Microsoft/TypeScript/issues/17112#issuecomment-326684693 summarizes the issues.
TypeScript Version: 2.4.1 / nightly (2.5.0-dev.20170707)
Code gulpfile.js
tsconfig.json
Expected behavior:
Actual behavior: Exception i got:
More
tsc
works fine.fp-ts
,fantasyland
andmonet