search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.2k stars 1.57k forks source link

HOT RELOAD: Flaky heap corruption when the VM is shutting down and a reload is triggered #26895

Closed johnmccutchan closed 8 years ago

johnmccutchan commented 8 years ago

We had flaky crashes on the build bots:

The above are two examples but the crash affected many different tests infrequently.

Based on stack traces it appeared to be an issue of reloading happening while the VM shutdown. I have successfully worked around this issue by landing this CL.

We need to determine the root cause of the crash.

johnmccutchan commented 8 years ago

It appears I have not successfully worked around this issue. It's still happening on the bots:

FAILED: none-vm release_x64 standalone/io/http_basic_test
Expected: Pass 
Actual: Crash
CommandOutput[vm]:

Command[vm]: DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --hot-reload-test-mode --ignore-unrecognized-flags --package-root=out/ReleaseX64/packages/ /b/build/slave/vm-linux-release-x64-live-reload-be/build/sdk/tests/standalone/io/http_basic_test.dart
Took 0:00:05.139734

Short reproduction command (experimental):
    python tools/test.py -mrelease --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --hot-reload --builder-tag no_ipv6 -t60 standalone/io/http_basic_test

FAILED: none-vm-checked release_x64 standalone/io/http_basic_test
Expected: Pass 
Actual: Crash
CommandOutput[vm]:

Command[vm]: DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --enable_asserts --enable_type_checks --hot-reload-test-mode --ignore-unrecognized-flags --package-root=out/ReleaseX64/packages/ /b/build/slave/vm-linux-release-x64-live-reload-be/build/sdk/tests/standalone/io/http_basic_test.dart
Took 0:00:05.523535

Short reproduction command (experimental):
    python tools/test.py -mrelease --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --hot-reload --builder-tag no_ipv6 --checked -t60 standalone/io/http_basic_test
fsc8000 commented 8 years ago

Backtrace from core dump:

(gdb) thread apply all bt

Thread 9 (Thread 0x7fef9f9c1780 (LWP 18707)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000922da1 in dart::Monitor::WaitMicros (this=0x1a43790, micros=0) at runtime/vm/os_thread_linux.cc:418
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x1a43790, millis=0) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fffeb0c5850, millis=0) at runtime/vm/lockers.h:169
#4  0x0000000000752fcd in dart::Dart::WaitForApplicationIsolateShutdown () at runtime/vm/dart.cc:321
#5  0x000000000075334c in dart::Dart::Cleanup () at runtime/vm/dart.cc:398
#6  0x00000000006896c9 in dart::Dart_Cleanup () at runtime/vm/dart_api_impl.cc:1180
#7  0x000000000067b168 in dart::bin::main (argc=6, argv=0x7fffeb0c5af8) at runtime/bin/main.cc:1720
#8  0x000000000067b283 in main (argc=6, argv=0x7fffeb0c5af8) at runtime/bin/main.cc:1752

Thread 8 (Thread 0x7fef9f9bf700 (LWP 18709)):
#0  0x00007fef9e6b5a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000006d1a14 in dart::bin::EventHandlerImplementation::Poll (args=27538048) at runtime/bin/eventhandler_linux.cc:391
#2  0x00000000006cc372 in dart::bin::ThreadStart (data_ptr=0x1a43160) at runtime/bin/thread_linux.cc:91
#3  0x00007fef9f5b4184 in start_thread (arg=0x7fef9f9bf700) at pthread_create.c:312
#4  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fef85b99700 (LWP 18752)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000000000922e65 in dart::Monitor::WaitMicros (this=0x7fef90129f80, micros=5000000) at runtime/vm/os_thread_linux.cc:423
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x7fef90129f80, millis=5000) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fef85b98dc0, millis=5000) at runtime/vm/lockers.h:169
#4  0x0000000000a0e521 in dart::ThreadPool::Worker::Loop (this=0x7fef90129f80) at runtime/vm/thread_pool.cc:420
#5  0x0000000000a0e6e9 in dart::ThreadPool::Worker::Main (args=140666891050880) at runtime/vm/thread_pool.cc:464
#6  0x0000000000921416 in dart::ThreadStart (data_ptr=0x7fef90036d70) at runtime/vm/os_thread_linux.cc:123
#7  0x00007fef9f5b4184 in start_thread (arg=0x7fef85b99700) at pthread_create.c:312
#8  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7fef85c9a700 (LWP 18751)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000922da1 in dart::Monitor::WaitMicros (this=0x7fef900103b0, micros=0) at runtime/vm/os_thread_linux.cc:418
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x7fef900103b0, millis=0) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fef85c994b0, millis=0) at runtime/vm/lockers.h:169
#4  0x000000000073fda6 in dart::BackgroundCompiler::Run (this=0x7fef90010350) at runtime/vm/compiler.cc:1894
#5  0x0000000000a0e3e4 in dart::ThreadPool::Worker::Loop (this=0x7fef90015ff0) at runtime/vm/thread_pool.cc:407
#6  0x0000000000a0e6e9 in dart::ThreadPool::Worker::Main (args=140666889920496) at runtime/vm/thread_pool.cc:464
#7  0x0000000000921416 in dart::ThreadStart (data_ptr=0x7fef90133020) at runtime/vm/os_thread_linux.cc:123
#8  0x00007fef9f5b4184 in start_thread (arg=0x7fef85c9a700) at pthread_create.c:312
#9  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7fef876c2700 (LWP 18750)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000000000922e65 in dart::Monitor::WaitMicros (this=0x7fef90010470, micros=5000000) at runtime/vm/os_thread_linux.cc:423
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x7fef90010470, millis=5000) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fef876c1dc0, millis=5000) at runtime/vm/lockers.h:169
#4  0x0000000000a0e521 in dart::ThreadPool::Worker::Loop (this=0x7fef90010470) at runtime/vm/thread_pool.cc:420
#5  0x0000000000a0e6e9 in dart::ThreadPool::Worker::Main (args=140666889897072) at runtime/vm/thread_pool.cc:464
#6  0x0000000000921416 in dart::ThreadStart (data_ptr=0x7fef90010510) at runtime/vm/os_thread_linux.cc:123
#7  0x00007fef9f5b4184 in start_thread (arg=0x7fef876c2700) at pthread_create.c:312
#8  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fef87cc4700 (LWP 18748)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000000000922e65 in dart::Monitor::WaitMicros (this=0x7fef90036730, micros=5000000) at runtime/vm/os_thread_linux.cc:423
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x7fef90036730, millis=5000) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fef87cc3dc0, millis=5000) at runtime/vm/lockers.h:169
#4  0x0000000000a0e521 in dart::ThreadPool::Worker::Loop (this=0x7fef90036730) at runtime/vm/thread_pool.cc:420
#5  0x0000000000a0e6e9 in dart::ThreadPool::Worker::Main (args=140666890053424) at runtime/vm/thread_pool.cc:464
#6  0x0000000000921416 in dart::ThreadStart (data_ptr=0x7fef900367d0) at runtime/vm/os_thread_linux.cc:123
#7  0x00007fef9f5b4184 in start_thread (arg=0x7fef87cc4700) at pthread_create.c:312
#8  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fef85075700 (LWP 18797)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000000000922e65 in dart::Monitor::WaitMicros (this=0x7fef800894d0, micros=5000000) at runtime/vm/os_thread_linux.cc:423
#2  0x0000000000922cbe in dart::Monitor::Wait (this=0x7fef800894d0, millis=5000) at runtime/vm/os_thread_linux.cc:402
#3  0x00000000006a9bf3 in dart::MonitorLocker::Wait (this=0x7fef85074dc0, millis=5000) at runtime/vm/lockers.h:169
#4  0x0000000000a0e521 in dart::ThreadPool::Worker::Loop (this=0x7fef800894d0) at runtime/vm/thread_pool.cc:420
#5  0x0000000000a0e6e9 in dart::ThreadPool::Worker::Main (args=140666621957328) at runtime/vm/thread_pool.cc:464
#6  0x0000000000921416 in dart::ThreadStart (data_ptr=0x7fef80089570) at runtime/vm/os_thread_linux.cc:123
#7  0x00007fef9f5b4184 in start_thread (arg=0x7fef85075700) at pthread_create.c:312
#8  0x00007fef9e6b537d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fef9c447700 (LWP 18800)):
#0  dart::BaseThread::is_os_thread (this=0x7fef80022a50) at runtime/vm/os_thread.h:37
#1  0x00000000006a759a in dart::Thread::Current () at runtime/vm/thread.h:157
#2  0x00000000006e90c8 in dart::StackResource::Init (this=0x7fef9c444940, thread=0x7fef80022a50) at runtime/vm/allocation.cc:60
#3  0x00000000006a7296 in dart::StackResource::StackResource (this=0x7fef9c444940, thread=0x7fef80022a50) at runtime/vm/allocation.h:41
#4  0x00000000006e9207 in dart::NoSafepointScope::NoSafepointScope (this=0x7fef9c444940) at runtime/vm/allocation.cc:79
#5  0x00000000008f2bd9 in dart::TypedData::GetInt8 (this=0x7fef6012bab0, byte_offset=4) at runtime/vm/object.h:7591
#6  0x00000000008bde53 in dart::ObjectPool::InfoAt (this=0x7fef6012ba10, index=4) at runtime/vm/object.cc:11539
#7  0x000000000091196d in dart::Code::ResetICDatas (this=0x7fef600886e0) at runtime/vm/object_reload.cc:93
#8  0x0000000000872535 in dart::MarkFunctionsForRecompilation::PreserveUnoptimizedCode (this=0x7fef9c444ba0) at runtime/vm/isolate_reload.cc:941
#9  0x000000000087244d in dart::MarkFunctionsForRecompilation::VisitObject (this=0x7fef9c444ba0, obj=0x7fef870b0ea1) at runtime/vm/isolate_reload.cc:917
#10 0x00000000009234fa in dart::HeapPage::VisitObjects (this=0x7fef87096000, visitor=0x7fef9c444ba0) at runtime/vm/pages.cc:84
#11 0x0000000000924dea in dart::PageSpace::VisitObjects (this=0x7fef80021388, visitor=0x7fef9c444ba0) at runtime/vm/pages.cc:606
#12 0x00000000007fd3b7 in dart::Heap::VisitObjects (this=0x7fef80021270, visitor=0x7fef9c444ba0) at runtime/vm/heap.cc:206
#13 0x0000000000870e54 in dart::IsolateReloadContext::MarkAllFunctionsForRecompilation (this=0x7fef60073cb0) at runtime/vm/isolate_reload.cc:964
#14 0x0000000000870eef in dart::IsolateReloadContext::InvalidateWorld (this=0x7fef60073cb0) at runtime/vm/isolate_reload.cc:973
#15 0x000000000087056b in dart::IsolateReloadContext::PostCommit (this=0x7fef60073cb0) at runtime/vm/isolate_reload.cc:739
#16 0x000000000086e88c in dart::IsolateReloadContext::FinishReload (this=0x7fef60073cb0) at runtime/vm/isolate_reload.cc:321
#17 0x000000000086513a in dart::Isolate::DoneFinalizing (this=0x7fef80000b10) at runtime/vm/isolate.cc:1103
#18 0x00000000006a5076 in dart::Dart_FinalizeLoading (complete_futures=true) at runtime/vm/dart_api_impl.cc:5697
#19 0x00000000006cb91d in dart::bin::Loader::LibraryTagHandler (tag=Dart_kScriptTag, library=0x1afadf8, url=0x7fef6010da88) at runtime/bin/loader.cc:616
#20 0x000000000086e5b4 in dart::IsolateReloadContext::StartReload (this=0x7fef60073cb0) at runtime/vm/isolate_reload.cc:281
#21 0x00000000008650fe in dart::Isolate::ReloadSources (this=0x7fef80000b10, test_mode=false) at runtime/vm/isolate.cc:1097
#22 0x0000000000730d6c in dart::DRT_HelperStackOverflow (isolate=0x7fef80000b10, thread=0x7fef80022a50, zone=0x7fef9c445328, arguments=...) at runtime/vm/code_generator.cc:1363
#23 0x0000000000730817 in dart::DRT_StackOverflow (arguments=...) at runtime/vm/code_generator.cc:1274
#24 0x00007fef9f83f430 in ?? ()
#25 0x00007fef80022a50 in ?? ()
#26 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fef9c832700 (LWP 18724)):
#0  0x000000000086b9b2 in dart::Isolate::origin_id (this=0xabababababababab) at runtime/vm/isolate.h:201
#1  0x00000000009daab0 in dart::ServiceIsolate::IsServiceIsolateDescendant (isolate=0xabababababababab) at runtime/vm/service_isolate.cc:122
#2  0x00000000009d98a1 in dart::ServiceEvent::ServiceEvent (this=0x7fef9c82fd00, isolate=0xabababababababab, event_kind=dart::ServiceEvent::kIsolateReload) at runtime/vm/service_event.cc:36
#3  0x000000000086e217 in dart::IsolateReloadContext::ReportError (this=0x7fef8802e8e0, error=...) at runtime/vm/isolate_reload.cc:214
#4  0x000000000086e641 in dart::IsolateReloadContext::StartReload (this=0x7fef8802e8e0) at runtime/vm/isolate_reload.cc:285
#5  0x00000000008650fe in dart::Isolate::ReloadSources (this=0x7fef78000b10, test_mode=false) at runtime/vm/isolate.cc:1097
#6  0x0000000000730d6c in dart::DRT_HelperStackOverflow (isolate=0x7fef78000b10, thread=0x7fef78022a50, zone=0x7fef9c8301d8, arguments=...) at runtime/vm/code_generator.cc:1363
#7  0x0000000000730817 in dart::DRT_StackOverflow (arguments=...) at runtime/vm/code_generator.cc:1274
#8  0x00007fef9f83f430 in ?? ()
#9  0x00007fef78022a50 in ?? ()
#10 0x0000000000000000 in ?? ()
johnmccutchan commented 8 years ago

Other notes:

fsc8000 commented 8 years ago

Running with asan, it reports a heap use-after-free. I think I have a fix. Trying it...

=================================================================
==11321==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e00002eb91 at pc 0xd5056e bp 0x7f40ccb5ffb0 sp 0x7f40ccb5ffa8
WRITE of size 1 at 0x60e00002eb91 thread T9
==11321==AddressSanitizer: while reporting a bug found another one.Ignoring.
    #0 0xd5056d in dart::IsolateReloadContext::ReportError(dart::Error const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd5056d)
    #1 0xd51856 in dart::IsolateReloadContext::StartReload() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd51856)
    #2 0xd34fbf in dart::Isolate::ReloadSources(bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd34fbf)
    #3 0x892231 in dart::DRT_StackOverflow(dart::NativeArguments) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x892231)
    #4 0x7f40d40e442f (+0x42f)
0x60e00002eb91 is located 17 bytes inside of 160-byte region [0x60e00002eb80,0x60e00002ec20)
freed by thread T9 here:
    #0 0x5cd4f9 in operator delete(void*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5cd4f9)
    #1 0xd3515a in dart::Isolate::DoneFinalizing() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd3515a)
    #2 0x691af4 in Dart_FinalizeLoading (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x691af4)
    #3 0x70125a in dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x70125a)
    #4 0xd51568 in dart::IsolateReloadContext::StartReload() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd51568)
    #5 0xd34fbf in dart::Isolate::ReloadSources(bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd34fbf)
    #6 0x892231 in dart::DRT_StackOverflow(dart::NativeArguments) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x892231)
    #7 0x7f40d40e442f (+0x42f)
    #8 0x7f40cc894dff (+0x24dff)
    #9 0x7f40c989a559 (+0x1b559)
    #10 0x7f40c989977b (+0x1a77b)
    #11 0x7f40cbb49247 (+0x9247)
    #12 0x7f40cbb76b32 (+0x36b32)
    #13 0x7f40cbb7683e (+0x3683e)
    #14 0x7f40cbb765b4 (+0x365b4)
    #15 0x7f40cbb6cc64 (+0x2cc64)
    #16 0x7f40cbb756d1 (+0x356d1)
    #17 0x7f40cbb745ed (+0x345ed)
    #18 0x7f40cbb7448c (+0x3448c)
    #19 0x7f40cbb6cc64 (+0x2cc64)
    #20 0x7f40cbb741ab (+0x341ab)
    #21 0x7f40cbb73d0e (+0x33d0e)
    #22 0x7f40cbb73bcb (+0x33bcb)
    #23 0x7f40cbb6cc64 (+0x2cc64)
    #24 0x7f40cbb6e037 (+0x2e037)
    #25 0x7f40cbb63563 (+0x23563)
    #26 0x7f40d40e484a (+0x84a)
    #27 0x927385 in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x927385)
    #28 0x9269dc in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x9269dc)
    #29 0x93144d in dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x93144d)
previously allocated by thread T9 here:
    #0 0x5cd1f9 in operator new(unsigned long) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5cd1f9)
    #1 0xd34f8c in dart::Isolate::ReloadSources(bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd34f8c)
    #2 0x892231 in dart::DRT_StackOverflow(dart::NativeArguments) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x892231)
    #3 0x7f40d40e442f (+0x42f)
    #4 0x7f40cc894dff (+0x24dff)
    #5 0x7f40c989a559 (+0x1b559)
    #6 0x7f40c989977b (+0x1a77b)
    #7 0x7f40cbb49247 (+0x9247)
    #8 0x7f40cbb76b32 (+0x36b32)
    #9 0x7f40cbb7683e (+0x3683e)
    #10 0x7f40cbb765b4 (+0x365b4)
    #11 0x7f40cbb6cc64 (+0x2cc64)
    #12 0x7f40cbb756d1 (+0x356d1)
    #13 0x7f40cbb745ed (+0x345ed)
    #14 0x7f40cbb7448c (+0x3448c)
    #15 0x7f40cbb6cc64 (+0x2cc64)
    #16 0x7f40cbb741ab (+0x341ab)
    #17 0x7f40cbb73d0e (+0x33d0e)
    #18 0x7f40cbb73bcb (+0x33bcb)
    #19 0x7f40cbb6cc64 (+0x2cc64)
    #20 0x7f40cbb6e037 (+0x2e037)
    #21 0x7f40cbb63563 (+0x23563)
    #22 0x7f40d40e484a (+0x84a)
    #23 0x927385 in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x927385)
    #24 0x9269dc in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x9269dc)
    #25 0x93144d in dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x93144d)
    #26 0xd28281 in dart::IsolateMessageHandler::HandleMessage(dart::Message*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd28281)
    #27 0xdecdcf in dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xdecdcf)
    #28 0xdee37d in dart::MessageHandler::TaskCallback() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xdee37d)
    #29 0xdf0c47 in dart::MessageHandlerTask::Run() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/message_handler.cc:29
Thread T9 created by T2 here:
    #0 0x5bdf02 in pthread_create (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5bdf02)
    #1 0x1076939 in dart::OSThread::Start(char const*, void (*)(unsigned long), unsigned long) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/os_thread_linux.cc:143
    #2 0x14195d4 in dart::ThreadPool::Worker::StartThread() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:362
    #3 0x14190a1 in dart::ThreadPool::Run(dart::ThreadPool::Task*) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:70
    #4 0xdec693 in dart::MessageHandler::PostMessage(dart::Message*, bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xdec693)
    #5 0x11b8e32 in dart::PortMap::PostMessage(dart::Message*) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/port.cc:258
    #6 0x627669 in Dart_Post (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x627669)
    #7 0x6fdc06 in dart::bin::Loader::Init(char const*, char const*, char const*, char const*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x6fdc06)
    #8 0x700f6f in dart::bin::Loader::LibraryTagHandler(Dart_LibraryTag, _Dart_Handle*, _Dart_Handle*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x700f6f)
    #9 0xd51568 in dart::IsolateReloadContext::StartReload() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd51568)
    #10 0xd34fbf in dart::Isolate::ReloadSources(bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd34fbf)
    #11 0x892231 in dart::DRT_StackOverflow(dart::NativeArguments) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x892231)
    #12 0x7f40d40e442f (+0x42f)
    #13 0x7f40d2045866
    #14 0x7f40d20456a5
    #15 0x7f40d20454d1
    #16 0x7f40d205a614
    #17 0x7f40d20441a4
    #18 0x7f40d2059bf4
    #19 0x7f40d20441a4
    #20 0x7f40d20598eb
    #21 0x7f40d205944e
    #22 0x7f40d205930b
    #23 0x7f40d20441a4
    #24 0x7f40d20591b7
    #25 0x7f40d2048463
    #26 0x7f40d40e484a (+0x84a)
    #27 0x927385 in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x927385)
    #28 0x9269dc in dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x9269dc)
    #29 0x93144d in dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x93144d)
    #30 0xd28281 in dart::IsolateMessageHandler::HandleMessage(dart::Message*) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xd28281)
    #31 0xdecdcf in dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xdecdcf)
    #32 0xdee37d in dart::MessageHandler::TaskCallback() (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0xdee37d)
    #33 0xdf0c47 in dart::MessageHandlerTask::Run() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/message_handler.cc:29
    #34 0x141c666 in dart::ThreadPool::Worker::Loop() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:407
    #35 0x141bf2a in dart::ThreadPool::Worker::Main(unsigned long) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:464
    #36 0x1076e68 in dart::ThreadStart(void*) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/os_thread_linux.cc:123
    #37 0x5d3933 in __asan::AsanThread::ThreadStart(unsigned long) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5d3933)
Thread T2 created by T0 here:
    #0 0x5bdf02 in pthread_create (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5bdf02)
    #1 0x1076939 in dart::OSThread::Start(char const*, void (*)(unsigned long), unsigned long) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/os_thread_linux.cc:143
    #2 0x14195d4 in dart::ThreadPool::Worker::StartThread() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:362
    #3 0x14190a1 in dart::ThreadPool::Run(dart::ThreadPool::Task*) /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/thread_pool.cc:70
    #4 0x1357808 in dart::ServiceIsolate::Run() /usr/local/google/home/fschneider/dartgit2/sdk/runtime/vm/service_isolate.cc:458
    #5 0x90ccb1 in dart::Dart::InitOnce(unsigned char const*, unsigned char const*, unsigned char const*, _Dart_Isolate* (*)(char const*, char const*, char const*, char const*, Dart_IsolateFlags*, void*, char**), void (*)(void*), void (*)(), void* (*)(char const*, bool), void (*)(unsigned char const**, long*, void*), void (*)(void const*, long, void*), void (*)(void*), bool (*)(unsigned char*, long), _Dart_Handle* (*)()) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x90ccb1)
    #6 0x61a1f3 in Dart_Initialize (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x61a1f3)
    #7 0x5e57cf in dart::bin::main(int, char**) (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5e57cf)
    #8 0x5e6dfa in main (/usr/local/google/home/fschneider/dartgit2/sdk/out/DebugX64/dart+0x5e6dfa)
    #9 0x7f40d6751f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
SUMMARY: AddressSanitizer: heap-use-after-free ??:0 dart::IsolateReloadContext::ReportError(dart::Error const&)
Shadow bytes around the buggy address:
  0x0c1c7fffdd20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c7fffdd30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c7fffdd40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c7fffdd50: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fffdd60: 00 00 00 00 00 00 04 fa fa fa fa fa fa fa fa fa
=>0x0c1c7fffdd70: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1c7fffdd80: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
  0x0c1c7fffdd90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1c7fffdda0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c7fffddb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c7fffddc0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==11321==ABORTING
johnmccutchan commented 8 years ago

I believe this is fixed. Thanks @fsc8000