Closed adamalfredsson closed 3 years ago
Your diagnostics report was recorded when you have been running a single test file only. Does it work with a single test, and/or a specific subproject, or throws OOM exception too as when running the tests for all of your subprojects?
Can you please try
launchctl setenv NODE_OPTIONS --max_old_space_size=8192
instead of
launchctl setenv NODE_OPTIONS --max_old_space_size=4096
Also, if it doesn't help,
> node
> require('v8').getHeapStatistics().heap_size_limit / (1024 * 1024)
Thank you for investigating!
Yes, the diagnostics report was recorded when running a single test file, as it succeeded this way. When running the complete project and process runs out of memory, no diagnostics report is available.
I'm getting the same error after running launchctl setenv NODE_OPTIONS --max_old_space_size=8192
. Trying again I got a bit more detailed error, if that's to any help:
[Info] Finished executing 6457 affected test(s) in 39.11 second(s)
[Error] <--- Last few GCs --->
[Error] [16886:0x158008000] 199674 ms: Scavenge 1998.2 (2066.1) -> 1996.5 (2069.8) MB, 7.8 / 0.0 ms (average mu = 0.774, current mu = 0.385) allocation failure
[Error] [16886:0x158008000] 199721 ms: Scavenge 2001.7 (2069.8) -> 1999.5 (2088.3) MB, 11.7 / 0.0 ms (average mu = 0.774, current mu = 0.385) allocation failure
[Error] [16886:0x158008000] 200385 ms: Mark-sweep 2012.7 (2088.3) -> 2002.0 (2096.1) MB, 622.8 / 0.0 ms (average mu = 0.637, current mu = 0.235) allocation failure scavenge might not succeed
[Error] <--- JS stacktrace --->
[Error] FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
[Error] 1: 0x104318114 node::Abort() [/opt/homebrew/bin/node]
[Error] 2: 0x104318294 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string<std::nullptr_t>(char const*) [/opt/homebrew/bin/node]
[Error] 3: 0x10443f2e8 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/bin/node]
[Error] 4: 0x10443f27c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/bin/node]
[Error] 5: 0x104584b14 v8::internal::Heap::CollectionBarrier::ShutdownRequested() [/opt/homebrew/bin/node]
[Error] 6: 0x1045bb564 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/opt/homebrew/bin/node]
[Error] 7: 0x1045abe88 void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/opt/homebrew/bin/node]
[Error] 8: 0x1045abb2c v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/opt/homebrew/bin/node]
[Error] 9: 0x1045ab84c v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/opt/homebrew/bin/node]
[Error] 10: 0x1045be840 v8::internal::PageEvacuationTask::ProcessItems() [/opt/homebrew/bin/node]
[Error] 11: 0x1045be7c4 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/opt/homebrew/bin/node]
[Error] 12: 0x104596644 v8::internal::ItemParallelJob::Task::RunInternal() [/opt/homebrew/bin/node]
[Error] 13: 0x1045969a0 v8::internal::ItemParallelJob::Run() [/opt/homebrew/bin/node]
[Error] 14: 0x1045ad158 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/opt/homebrew/bin/node]
[Error] 15: 0x1045acf64 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/opt/homebrew/bin/node]
[Error] 16: 0x1045a1938 v8::internal::MarkCompactCollector::Evacuate() [/opt/homebrew/bin/node]
[Error] 17: 0x10459fc30 v8::internal::MarkCompactCollector::CollectGarbage() [/opt/homebrew/bin/node]
[Error] 18: 0x104585024 v8::internal::Heap::MarkCompact() [/opt/homebrew/bin/node]
[Error] 19: 0x104583364 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/homebrew/bin/node]
[Error] 20: 0x104581c00 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/bin/node]
[Error] 21: 0x10458a7a0 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/bin/node]
[Error] 22: 0x10458a808 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/bin/node]
[Error] 23: 0x1045628a4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [/opt/homebrew/bin/node]
[Error] 24: 0x104562660 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/opt/homebrew/bin/node]
[Error] 25: 0x10470ccc4 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType, v8::internal::MinimumCapacity) [/opt/homebrew/bin/node]
[Error] 26: 0x10470d1d0 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::AllocationType) [/opt/homebrew/bin/node]
[Error] 27: 0x10470dc84 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/opt/homebrew/bin/node]
[Error] 28: 0x10470dbcc v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/opt/homebrew/bin/node]
[Error] 29: 0x1047eecec v8::internal::Runtime_AddDictionaryProperty(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/bin/node]
[Error] 30: 0x104a61aec Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/opt/homebrew/bin/node]
[Error] 31: 0x285b3782038c
[Error] 32: 0x285b3778d1f0
[Error] 33: 0x1049f1e24 Builtins_ArgumentsAdaptorTrampoline [/opt/homebrew/bin/node]
[Error] 34: 0x285b3781c598
[Error] 35: 0x285b37821acc
[Error] 36: 0x285b378154e0
[Error] 37: 0x285b3778d1f0
[Error] 38: 0x1049f1e24 Builtins_ArgumentsAdaptorTrampoline [/opt/homebrew/bin/node]
[Error] 39: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 40: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 41: 0x285b37e5b720
[Error] 42: 0x1049f1e24 Builtins_ArgumentsAdaptorTrampoline [/opt/homebrew/bin/node]
[Error] 43: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 44: 0x1049f1e24 Builtins_ArgumentsAdaptorTrampoline [/opt/homebrew/bin/node]
[Error] 45: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 46: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 47: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 48: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 49: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 50: 0x285b389eb080
[Error] 51: 0x1049fa7f4 Builtins_InterpreterEntryTrampoline [/opt/homebrew/bin/node]
[Error] 52: 0x1049f80a4 Builtins_JSEntryTrampoline [/opt/homebrew/bin/node]
[Error] 53: 0x1049f7d48 Builtins_JSEntry [/opt/homebrew/bin/node]
[Error] 54: 0x158008000
Also perhaps worth pointing out is that I've tried configuring wallaby for arm64 environment. I've added this to my editor settings:
"wallaby.node": "/opt/homebrew/bin/node",
But I get the same result without this configuration. Also my colleague is running on x86_64 macOS with the same issue.
Another interesting point is that I've set tsconfig target to ES5
in tsconfig.wallaby.spec.json:
{
"extends": "./tsconfig.json",
"compilerOptions": {
"target": "es5",
"downlevelIteration": true,
"types": ["jasmine", "node"]
}
}
If I only set the target to ES2015
instead, I'm able to run all tests.
I'm also able to run roughly one third (2195/6457 tests) of all tests by configuring tests
in package.json
:
"wallaby": {
"tests": [
"projects/storefrontlib/**/*.spec.ts"
]
}
Also excluding these 2195 tests and running the other 4261 tests succeeds.
My impression is that despite running these commands to allocate more memory, it doesn't have effect. It still seems to fail on ~2048 Mb (my interpretation of Mark-sweep 2012.7 (2088.3) -> 2002.0 (2096.1) MB
). Perhaps ES2015 is just less memory-intensive?
Could you also please provide the results of this?
Also, if it doesn't help,
- can you try running the project with node v14.x?
- run (and share the result):
> node > require('v8').getHeapStatistics().heap_size_limit / (1024 * 1024)
Sorry for the delay, I had issues setting up NVM.
Could you also please provide the results of this?
Also, if it doesn't help,
- can you try running the project with node v14.x?
Running the project with arm64 node version 14.x runs into the same exception.
- run (and share the result):
> node > require('v8').getHeapStatistics().heap_size_limit / (1024 * 1024)
4144
Hey @nomadoda - sorry it's taken us a few days to get back to you. We've created a sample project with a large amount of tests to try reproduce your problem but haven't been able to do so yet. We're also in the process of analyzing the heap to see if there's anything suspicious going on but have not identified anything yet.
In terms of where you are at - are you OK to target ES2015
instead of ES5
as a workaround? If so, I'll close the issue. We'll continue to do a bit more investigation but I feel we might be wasting our time without having your actual project.
The fact that it's working using ES2015
and not ES5
indicates to us that they issue may lie in something that Webpack is doing as Webpack is the thing that's compiling your TypeScript, not Wallaby.
Another difference worth noting when running Wallaby vs. ng test
is that Wallaby is running all of your projects at the same time, in the same Webpack while ng test
runs them sequentially in their own process space, isolated from one another; this may also be contributing to the problem.
Do you have a problem if you try and run a single project? Or is it only when you run all projects at the same time?
I'd also be interested in upgrading to latest version of Angular (and your various dependencies) to see if that fixes the problem.
We're also in the process of analyzing the heap to see if there's anything suspicious going on but have not identified anything yet.
We completed our investigation today and didn't find anything that would lead to your issue. Everything was behaving as expected.
If you're able to run on a machine with enough memory, we're keen to understand if the problem goes away if you allocate enough memory (e.g. try 16GB).
Thank you a lot for investigating. I understand it's a real difficult one to look at. I did manage to reproduce this issue with another machine, but hadn't tried allocating more memory. Targeting ES2015
works fine, however some ~70 tests are incompatible and fails on ES2015
.
I have no issue running a single project, it's only when running all at one time.
I'll post an update when we upgrade angular as to whether it works.
I'm going to close this issue since we haven't heard back and it is old.
@nomadoda - feel free to reply when you have more details and we can re-open the issue. Alternatively, you may create a new issue if you're still having problems.
Issue description or question
I'm running wallaby in a fairly large project and after completes finish running the process runs out of memory and exits:
I have tried running command suggested in this issue and this issue:
And setting the universal variable for fish shell (probably irrelevant):
And launching vscode from terminal with NODE_OPTIONS set:
Unfortunately neither attempt solved this issue. Any ideas?
Wallaby diagnostics report
I'm able to retrieve the diagnostics report when running exclusive test run: