search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.09k stars 1.56k forks source link

language/vm/closure_memory_retention_test passes on my machine #28932

Open peter-ahe-google opened 7 years ago

peter-ahe-google commented 7 years ago

macOS Sierra, Version 10.12.3 (16D32).

  Model Name:   Mac Pro
  Model Identifier: MacPro6,1
  Processor Name:   12-Core Intel Xeon E5
  Processor Speed:  2.7 GHz
  Number of Processors: 1
  Total Number of Cores:    12
  L2 Cache (per Core):  256 KB
  L3 Cache: 30 MB
  Memory:   64 GB
FAILED: dartk-vm release_x64 language/vm/closure_memory_retention_test
Expected: Crash Timeout 
Actual: Pass
CommandOutput[vm]:

stderr:
Exhausted heap space, trying to allocate 1328 bytes.
Exhausted heap space, trying to allocate 384 bytes.

Command[vm]: DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --dfe=utils/kernel-service/kernel-service.dart --old_gen_heap_size=50 --ignore-unrecognized-flags --packages=/Users/ahe/w/dart-sdk/sdk/.packages /Users/ahe/w/dart-sdk/sdk/tests/language/vm/closure_memory_retention_test.dart
Took 0:00:04.289941

Short reproduction command (experimental):
    python tools/test.py -mrelease -cdartk -t60 language/vm/closure_memory_retention_test
jensjoha commented 7 years ago

This test is currently failing on vm-kernel-linux-debug-x64-be

The kernel expectation was previously "Crash, Timeout", though that was deleted in 628be2b36a760291c4cbfc57e5801bd47dca5f0f

Locally I can reproduce a "timeout" via

while [ 1 ]; do echo -n .; out/DebugX64/dart --dfe=out/DebugX64/gen/kernel-service.dart.snapshot --old_gen_heap_size=50 --no-enable-malloc-hooks --ignore-unrecognized-f lags --packages=.packages tests/language/vm/closure_memory_retention_test.dart; done

mostly it just runs and retries, but eventually (at least on my machine (this time at iteration 259)) it will print something like

Exhausted heap space, trying to allocate 1904 bytes.
../../runtime/vm/exceptions.cc: 133: error: expected: frame != NULL
Dumping native stack trace for thread f78c

and seemingly get stuck (while still using ~15% cpu).

Stacktrace via gdb --batch --quiet -ex "thread apply all bt" -ex "quit" -p {pid goes here}:

Unable to determine compiler version.
Skipping loading of libstdc++ pretty-printers for now.
Non-google3 binary detected.
[New LWP 63375]
[New LWP 63374]
[New LWP 63373]
[New LWP 63372]
[New LWP 63370]
[New LWP 63369]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/grte/v4/lib64/libthread_db.so.1".
Loading gdb's copy of v17 libstdc++ pretty-printers.
pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.

Thread 7 (Thread 0x7f8c79671700 (LWP 63369)):
#0  0x00007f8c78365a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000534c05 in dart::bin::EventHandlerImplementation::Poll (args=<optimized out>) at ../../runtime/bin/eventhandler_linux.cc:394
#2  0x000000000055951e in dart::bin::ThreadStart (data_ptr=<optimized out>) at ../../runtime/bin/thread_linux.cc:89
#3  0x00007f8c79060184 in start_thread (arg=0x7f8c79671700) at pthread_create.c:312
#4  0x00007f8c7836537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f8c7826a700 (LWP 63370)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x000000000092acbc in dart::Monitor::WaitMicros (this=0x1cba2a0, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:445
#2  0x0000000000a4e5e9 in WaitMicros (micros=137, this=<optimized out>) at ../../runtime/vm/thread_interrupter.cc:177
#3  dart::ThreadInterrupter::ThreadMain (parameters=<optimized out>) at ../../runtime/vm/thread_interrupter.cc:177
#4  0x0000000000929e21 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:125
#5  0x00007f8c79060184 in start_thread (arg=0x7f8c7826a700) at pthread_create.c:312
#6  0x00007f8c7836537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f8c77fa8700 (LWP 63372)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000092ace4 in dart::Monitor::WaitMicros (this=0x1cba7e0, micros=0) at ../../runtime/vm/os_thread_linux.cc:440
#2  0x0000000000859eb7 in dart::MonitorLocker::WaitWithSafepointCheck (this=0x7f8c77fa43b0, thread=0x1cd9b00, millis=0) at ../../runtime/vm/lockers.cc:35
#3  0x00000000007affce in dart::HeapIterationScope::HeapIterationScope (this=<optimized out>, writable=<optimized out>) at ../../runtime/vm/heap.cc:217
#4  0x00000000007b0391 in dart::Heap::FindOldObject (this=0x301a000, visitor=0x7f8c77fa4490) at ../../runtime/vm/heap.cc:284
#5  0x00000000008db16d in dart::Code::LookupCodeInIsolate (isolate=0x3008000, pc=10183283) at ../../runtime/vm/object.cc:14411
#6  0x00000000009b9066 in DumpStackFrame (frame_index=0, pc=<optimized out>) at ../../runtime/vm/profiler.cc:335
#7  dart::ProfilerStackWalker::Append (this=0x7f8c77fa4590, pc=10183283) at ../../runtime/vm/profiler.cc:386
#8  0x00000000009b65a3 in dart::ProfilerNativeStackWalker::walk (this=0x7f8c77fa4590) at ../../runtime/vm/profiler.cc:634
#9  0x00000000009b6240 in dart::Profiler::DumpStackTrace (sp=140241285039624, fp=140241285039680, pc=10183283) at ../../runtime/vm/profiler.cc:1031
#10 0x0000000000692a21 in dart::DynamicAssertionHelper::Fail (this=0x7f8c77fa5898, format=<optimized out>) at ../../runtime/platform/assert.cc:41
#11 0x000000000073776c in BuildStackTrace (builder=<optimized out>) at ../../runtime/vm/exceptions.cc:133
#12 dart::ThrowExceptionHelper (thread=0x1cd9b00, incoming_exception=..., existing_stacktrace=..., is_rethrow=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at ../../runtime/vm/exceptions.cc:574
#13 0x000000000073728d in dart::Exceptions::Throw (thread=0x1cd9b00, exception=...) at ../../runtime/vm/exceptions.cc:785
#14 0x000000000087a97b in dart::Object::Allocate (cls_id=70, size=1904, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:1881
#15 0x00000000008fd380 in dart::Array::New (class_id=70, len=234, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:21632
#16 0x00000000008828b7 in dart::Array::New (len=234, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:21621
#17 0x00000000007749ad in dart::FlowGraphCompiler::CreateDeoptInfo (this=0x7f8c77fa5f10, assembler=0x7f8c77fa6550) at ../../runtime/vm/flow_graph_compiler.cc:950
#18 0x00000000006f7cd4 in dart::CompileParsedFunctionHelper::FinalizeCompilation (this=0x7f8c77fa7180, assembler=0x7f8c77fa6550, graph_compiler=<optimized out>, flow_graph=0x383c9f8) at ../../runtime/vm/compiler.cc:537
#19 0x00000000006faa34 in dart::CompileParsedFunctionHelper::Compile (this=<optimized out>, pipeline=<optimized out>) at ../../runtime/vm/compiler.cc:1154
#20 0x00000000006fbd73 in dart::CompileFunctionHelper (pipeline=<optimized out>, function=..., optimized=<optimized out>, osr_id=-1) at ../../runtime/vm/compiler.cc:1265
#21 0x00000000006fcbf9 in dart::Compiler::CompileOptimizedFunction (thread=<optimized out>, function=..., osr_id=-1) at ../../runtime/vm/compiler.cc:1533
#22 0x00000000006fea0b in dart::BackgroundCompiler::Run (this=<optimized out>) at ../../runtime/vm/compiler.cc:1975
#23 0x0000000000a4fb9c in dart::ThreadPool::Worker::Loop (this=<optimized out>) at ../../runtime/vm/thread_pool.cc:402
#24 0x0000000000a4f9e7 in dart::ThreadPool::Worker::Main (args=50561344) at ../../runtime/vm/thread_pool.cc:459
#25 0x0000000000929e21 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:125
#26 0x00007f8c79060184 in start_thread (arg=0x7f8c77fa8700) at pthread_create.c:312
#27 0x00007f8c7836537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f8c7616d700 (LWP 63373)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000092ace4 in dart::Monitor::WaitMicros (this=0x1cbade0, micros=0) at ../../runtime/vm/os_thread_linux.cc:440
#2  0x00000000006febfe in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::BackgroundCompiler::Run (this=<optimized out>) at ../../runtime/vm/compiler.cc:2013
#4  0x0000000000a4fb9c in dart::ThreadPool::Worker::Loop (this=<optimized out>) at ../../runtime/vm/thread_pool.cc:402
#5  0x0000000000a4f9e7 in dart::ThreadPool::Worker::Main (args=50562144) at ../../runtime/vm/thread_pool.cc:459
#6  0x0000000000929e21 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:125
#7  0x00007f8c79060184 in start_thread (arg=0x7f8c7616d700) at pthread_create.c:312
#8  0x00007f8c7836537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f8c75ef9700 (LWP 63374)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000092ace4 in dart::Monitor::WaitMicros (this=0x1cbaae0, micros=0) at ../../runtime/vm/os_thread_linux.cc:440
#2  0x0000000000a069ac in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::SafepointHandler::ExitSafepointUsingLock (this=<optimized out>, T=<optimized out>) at ../../runtime/vm/safepoint.cc:174
#4  0x0000000000859ee3 in dart::MonitorLocker::WaitWithSafepointCheck (this=0x7f8c75ef4228, thread=0x1cd8900, millis=<optimized out>) at ../../runtime/vm/lockers.cc:47
#5  0x000000000092dcdd in dart::PageSpace::MarkSweep (this=<optimized out>, invoke_api_callbacks=<optimized out>) at ../../runtime/vm/pages.cc:856
#6  0x00000000007b122d in dart::Heap::CollectOldSpaceGarbage (this=<optimized out>, thread=<optimized out>, api_callbacks=dart::Heap::kInvokeApiCallbacks, reason=<optimized out>) at ../../runtime/vm/heap.cc:402
#7  0x00000000007afb4d in dart::Heap::CollectAllGarbage (this=0x301ad00) at ../../runtime/vm/heap.cc:448
#8  0x00000000007af8d4 in dart::Heap::AllocateOld (this=<optimized out>, size=<optimized out>, type=<optimized out>) at ../../runtime/vm/heap.cc:98
#9  0x0000000000877a6f in dart::Heap::Allocate (this=0x301ad00, size=48, space=dart::Heap::kOld) at ../../runtime/vm/heap.h:67
#10 0x000000000087a8f3 in dart::Object::Allocate (cls_id=70, size=48, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:1875
#11 0x00000000008fd380 in dart::Array::New (class_id=70, len=2, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:21632
#12 0x00000000008828b7 in dart::Array::New (len=2, space=dart::Heap::kOld) at ../../runtime/vm/object.cc:21621
#13 0x000000000077130a in dart::FlowGraphCompiler::InitCompiler (this=<optimized out>) at ../../runtime/vm/flow_graph_compiler.cc:282
#14 0x000000000077e46c in dart::FlowGraphCompiler::CompileGraph (this=0x7f8c75ef4a80) at ../../runtime/vm/flow_graph_compiler_x64.cc:967
#15 0x00000000006fa971 in dart::CompileParsedFunctionHelper::Compile (this=<optimized out>, pipeline=<optimized out>) at ../../runtime/vm/compiler.cc:1131
#16 0x00000000006fdf03 in dart::Compiler::EvaluateStaticInitializer (field=...) at ../../runtime/vm/compiler.cc:1707
#17 0x00000000008baac1 in dart::Field::EvaluateInitializer (this=0x7f8c75ef6308) at ../../runtime/vm/object.cc:8003
#18 0x00000000006f0cde in DRT_HelperInitStaticField (isolate=<optimized out>, thread=<optimized out>, zone=<optimized out>, arguments=...) at ../../runtime/vm/code_generator.cc:2249
#19 dart::DRT_InitStaticField (arguments=...) at ../../runtime/vm/code_generator.cc:2247
#20 0x00007f8c794f1630 in ?? ()
#21 0x0000000001cd8900 in ?? ()
#22 0x0000000000000001 in ?? ()
#23 0x00007f8c75ef6828 in ?? ()
#24 0x00007f8c75ef6830 in ?? ()
#25 0xabababababababab in ?? ()
#26 0x00007f8c76a83fb1 in ?? ()
#27 0x00007f8c795346e1 in ?? ()
#28 0x00007f8c75ef6858 in ?? ()
#29 0x00007f8c774e45e2 in ?? ()
#30 0x00007f8c76cffaa1 in ?? ()
#31 0x00007f8c79531021 in ?? ()
#32 0x00007f8c769e1b81 in ?? ()
#33 0x00007f8c79531021 in ?? ()
#34 0x00007f8c76a84051 in ?? ()
#35 0x00007f8c76c456f1 in ?? ()
#36 0x00007f8c75ef6898 in ?? ()
#37 0x00007f8c774e4842 in ?? ()
#38 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f8c758df700 (LWP 63375)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000092ace4 in dart::Monitor::WaitMicros (this=0x1cbb560, micros=0) at ../../runtime/vm/os_thread_linux.cc:440
#2  0x0000000000a06b9e in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::SafepointHandler::BlockForSafepoint (this=<optimized out>, T=<optimized out>) at ../../runtime/vm/safepoint.cc:193
#4  0x0000000000a98842 in CheckForSafepoint (this=<optimized out>) at ../../runtime/vm/thread.h:658
#5  dart::SweeperTask::Run (this=<optimized out>) at ../../runtime/vm/gc_sweeper.cc:134
#6  0x0000000000a4fb9c in dart::ThreadPool::Worker::Loop (this=<optimized out>) at ../../runtime/vm/thread_pool.cc:402
#7  0x0000000000a4f9e7 in dart::ThreadPool::Worker::Main (args=50562464) at ../../runtime/vm/thread_pool.cc:459
#8  0x0000000000929e21 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:125
#9  0x00007f8c79060184 in start_thread (arg=0x7f8c758df700) at pthread_create.c:312
#10 0x00007f8c7836537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f8c79673780 (LWP 63368)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000092ace4 in dart::Monitor::WaitMicros (this=0x1cbae40, micros=0) at ../../runtime/vm/os_thread_linux.cc:440
#2  0x000000000085750b in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::KernelCompilationRequest::SendAndWaitForResponse (this=<optimized out>, kernel_port=<optimized out>, script_uri=<optimized out>) at ../../runtime/vm/kernel_isolate.cc:334
#4  0x000000000085734b in dart::KernelIsolate::CompileToKernel (script_uri=<optimized out>) at ../../runtime/vm/kernel_isolate.cc:450
#5  0x00000000006827fe in Dart_CompileToKernel (script_uri=0x80 <error: Cannot access memory at address 0x80>) at ../../runtime/vm/dart_api_impl.cc:6031
#6  0x000000000052fc1c in dart::bin::CreateIsolateAndSetupHelper (is_main_isolate=<optimized out>, script_uri=0x7ffc88e70f6f "tests/language/vm/closure_memory_retention_test.dart", main=0xd8532b "main", package_root=0x0, packages_config=0x7ffc88e70f65 ".packages", flags=0x0, error=<optimized out>, exit_code=0x7ffc88e6ec2c) at ../../runtime/bin/main.cc:852
#7  0x000000000052f2bd in dart::bin::RunMainIsolate (script_name=0x7ffc88e70f6f "tests/language/vm/closure_memory_retention_test.dart", dart_options=0x7ffc88e6ecf8) at ../../runtime/bin/main.cc:1244
#8  0x0000000000530992 in dart::bin::main (argc=<optimized out>, argv=<optimized out>) at ../../runtime/bin/main.cc:1633
#9  0x0000000000530e99 in main (argc=30125676, argv=0x80) at ../../runtime/bin/main.cc:1674
A debugging session is active.

    Inferior 1 [process 63368] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]

So it seems that because it's in debug mode it tries to do a stacktrace because of a failed ASSERT which then seemingly waits forever for some safepoint. (why the assert fails etc I don't know --- maybe failed assumptions when dealing with oom? --- and why oom I don't know)

peter-ahe-google commented 7 years ago

Can we temporarily skip this test? I sometimes fix a lot of crashes and I forget issues like this.

chloestefantsova commented 7 years ago

It is currently skipped for dartkp. I created a CL to skip it for both dartk and dartkp.

peter-ahe-google commented 7 years ago

Thank you @stefantsov.

mraleph commented 7 years ago

The test seems to be failing because it is setting --old_gen_heap_size=50 globally which is apparently too small for front-end isolate to operate. We need to extend Dart_IsolateFlags to allow setting custom heap size for the DFE isolate - this should allow us to run this test without any issues.

@ErikCorryGoogle could you take a look?