Heap corruption crashes

[mraleph]

Can't repro locally but might be worth investigating (GC problem?)

FAILED: none-vm-checked debug_x64 standalone/io/http_proxy_test Expected: Pass Actual: Crash CommandOutput[vm]:

stderr: /Volumes/data/b/build/slave/vm-mac-debug-x64-be/build/dart/runtime/vm/raw_object.cc:210: error: expected: (instance_size == SizeTag::decode(tags)) || (SizeTag::decode(tags) == 0)

Command[vm]: xcodebuild/DebugX64/dart --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=xcodebuild/DebugX64/packages/ /Volumes/data/b/build/slave/vm-mac-debug-x64-be/build/dart/tests/standalone/io/http_proxy_test.dart Took 0:00:07.193000

Short reproduction command (experimental):     python tools/test.py -ax64 --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t120 standalone/io/http_proxy_test

[kodandersson]

Similar failure, this time on debug_simarm, and in a simpler test (but lots of GC activity):

FAILED: none-vm-checked debug_simarm standalone/verified_mem_test Expected: Pass Actual: Crash CommandOutput[vm]:

stderr: Verifying before marking... done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. runtime/vm/raw_object.cc:210: error: expected: (instance_size == SizeTag::decode(tags)) || (SizeTag::decode(tags) == 0)

Command[vm]: out/DebugSIMARM/dart --verified_mem --verify_before_gc --verify_after_gc --old_gen_growth_rate=1 --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=out/DebugSIMARM/packages/ /mnt/data/b/build/slave/vm-arm-sim-debug-be/build/dart/tests/standalone/verified_mem_test.dart Took 0:00:01.104000

Short reproduction command (experimental):     python tools/test.py -asimarm --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t480 standalone/verified_mem_test

---

cc @iposva-google. Changed the title to: "Heap corruption crashes".

[kodandersson]

Another one. Seems to affect all platforms and archs.

http://build.chromium.org/p/client.dart/builders/vm-win32-debug-russian-be/builds/4118/steps/checked_tests/logs/stdio

FAILED: none-vm-checked debug_ia32 language/disassemble_test Expected: Pass Actual: Crash CommandOutput[vm]:

stdout: Code for function 'dart:builtin::__getPrintClosure@221999692' { 01122700 bf79165303 mov edi,0x3531679 'Function '_getPrintClosure@221999692': static.' 01122705 ff472f inc [edi+0x2f] 01122708 817f2f28230000 cmp [edi+0x2f],0x2328 0112270F 0f8d6bf8ffff jnl 0x1121f80 [stub: OptimizeFunction] ... 03F713CC 58 pop eax 03F713CD 8945ec mov [ebp-0x14],eax 03F713D0 50 push eax         ;; GuardFieldClass:14(_used@709387912 <not-nullable _Smi@915557746>, t1) 03F713D1 58 pop eax

stderr: E:\b\build\slave\vm-win32-debug-russian-be\build\dart\runtime\vm/class_table.h:132: error: expected: IsValidIndex(index)

Command[vm]: build\DebugIA32\dart.exe --disassemble --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=build/DebugIA32/packages/ E:\b\build\slave\vm-win32-debug-russian-be\build\dart\tests\language\disassemble_test.dart Took 0:00:14.337000

Short reproduction command (experimental):     python tools/test.py --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t120 language/disassemble_test

[kodandersson]

Just observed this on my Linux desktop when sync'ed at r43479:

$ python tools/test.py --report --time --mode=release --arch=simmips,simarm --compiler=none --runtime=vm --failure-summary --write- debug-log --write-test-outcome-log --copy-coredumps --exclude-suite=pkg --checked Test configurations: none_vm_release_simmips_checked none_vm_release_simarm_checked [00:19 | --% | + 1743 | - 0]Total: 31892 tests   7351 tests will be skipped (7002 skipped by design)   18 tests are expected to be flaky but not crash   2 tests are expected to flaky crash   24217 tests are expected to pass   60 tests are expected to fail that we won't fix   234 tests are expected to fail that we should fix   10 tests are expected to crash that we should fix   0 tests are allowed to timeout   0 tests are skipped on browsers due to compile-time error   0 could not be categorized or are in multiple categories

[05:42 | 91% | +22537 | - 0] FAILED: none-vm-checked release_simarm language/cast_test/10 Expected: Pass Actual: Crash Runtime error expected. CommandOutput[vm]:

Command[vm]: out/ReleaseSIMARM/dart --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=out/ReleaseSIMARM/packages/ /usr/local/google/home/koda/flake/dart/out/ReleaseSIMMIPS/generated_tests/language/cast_test_10.dart Took 0:00:00.246000

Short reproduction command (experimental):     python tools/test.py -mrelease -asimarm --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t240 language/cast_test/10

[07:12 | 100% | +24540 | - 1] === Failure summary:

FAILED: none-vm-checked release_simarm language/cast_test/10 Expected: Pass Actual: Crash Runtime error expected. CommandOutput[vm]:

Command[vm]: out/ReleaseSIMARM/dart --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=out/ReleaseSIMARM/packages/ /usr/local/google/home/koda/flake/dart/out/ReleaseSIMMIPS/generated_tests/language/cast_test_10.dart Took 0:00:00.246000

Short reproduction command (experimental):     python tools/test.py -mrelease -asimarm --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t240 language/cast_test/10

=== === 1 test failed ===

[07:12 | 100% | +24540 | - 1]

--- Total time: 07:12 --- 0:02:15.408000 - vm - none-vm-checked release_simmips/lib/convert/utf85_test 0:01:43.601000 - vm - none-vm-checked release_simarm/corelib/int_parse_radix_test/02 0:01:41.837000 - vm - none-vm-checked release_simarm/corelib/int_parse_radix_test/none 0:01:39.887000 - vm - none-vm-checked release_simarm/corelib/int_parse_radix_test/01 0:01:17.126000 - vm - none-vm-checked release_simarm/corelib/big_integer_parsed_mul_div_vm_test 0:01:14.725000 - vm - none-vm-checked release_simmips/corelib/int_parse_radix_test/02 0:01:13.083000 - vm - none-vm-checked release_simmips/corelib/int_parse_radix_test/01 0:01:12.995000 - vm - none-vm-checked release_simmips/corelib/int_parse_radix_test/none 0:01:07.266000 - vm - none-vm-checked release_simmips/lib/convert/chunked_conversion_utf88_test 0:01:01.411000 - vm - none-vm-checked release_simarm/co19/LibTest/core/Uri/encodeQueryComponent_A01_t02 0:00:59.537000 - vm - none-vm-checked release_simmips/corelib/big_integer_parsed_mul_div_vm_test 0:00:58.780000 - vm - none-vm-checked release_simarm/lib/convert/streamed_conversion_json_utf8_decode_test 0:00:57.485000 - vm - none-vm-checked release_simarm/lib/convert/streamed_conversion_json_utf8_decode_test 0:00:49.642000 - vm - none-vm-checked release_simarm/lib/mirrors/mirrors_reader_test 0:00:46.495000 - vm - none-vm-checked release_simarm/lib/convert/json_utf8_chunk_test 0:00:45.372000 - vm - none-vm-checked release_simmips/co19/LibTest/core/Uri/encodeQueryComponent_A01_t02 0:00:44.588000 - vm - none-vm-checked release_simmips/lib/convert/streamed_conversion_json_utf8_decode_test 0:00:43.513000 - vm - none-vm-checked release_simarm/corelib/collection_length_test 0:00:41.875000 - vm - none-vm-checked release_simmips/lib/convert/streamed_conversion_json_utf8_decode_test 0:00:38.932000 - vm - none-vm-checked release_simarm/isolate/mandel_isolate_test

[kodandersson]

I produced a core dump and am debugging it now. This particular crash is during compilation, but seems like a random corruption of new'ed memory (a Redirection object, in this case).

Core was generated by `out/ReleaseSIMMIPS/dart --ignore-unrecognized-flags --enable_asserts --enable_t'. Program terminated with signal SIGSEGV, Segmentation fault.

­0 Get (argument_count=argument_count@entry=2, call_kind=dart::Simulator::kRuntimeCall, external_function=138550464) at runtime/vm/simulator_mips.cc:767

767 if (current->externalfunction == external_function) return current; (gdb) bt

­0 Get (argument_count=argument_count@entry=2, call_kind=dart::Simulator::kRuntimeCall, external_function=138550464) at runtime/vm/simulator_mips.cc:767

­1 dart::Simulator::RedirectExternalReference (function=138550464, call_kind=dart::Simulator::kRuntimeCall, argument_count=argument_count@entry=2)

    at runtime/vm/simulator_mips.cc:817

­2 0x0843e956 in dart::RuntimeEntry::Call (this=0x8bb1118 <dart::kUpdateFieldCidRuntimeEntry>, assembler=0xf653b370, argument_count=2)

    at runtime/vm/runtime_entry_mips.cc:41

­3 0x083ffa85 in dart::Assembler::CallRuntime (this=<optimized out>, entry=..., argument_count=argument_count@entry=2) at runtime/vm/assembler_mips.cc:971

­4 0x082ab9ad in dart::GuardFieldLengthInstr::EmitNativeCode (this=0xf5d12600, compiler=0xf653b470) at runtime/vm/intermediate_language_mips.cc:1821

­5 0x0823758c in dart::FlowGraphCompiler::VisitBlocks (this=this@entry=0xf653b470) at runtime/vm/flow_graph_compiler.cc:420

­6 0x0823c6c4 in dart::FlowGraphCompiler::CompileGraph (this=this@entry=0xf653b470) at runtime/vm/flow_graph_compiler_mips.cc:1144

­7 0x081d2210 in dart::CompileParsedFunctionHelper (pipeline=0xf653bf20, parsed_function=0xf653b850, optimized=false, osr_id=-1)

    at runtime/vm/compiler.cc:696

­8 0x081d3d33 in dart::CompileFunctionHelper (pipeline=0xf653bf20, function=..., optimized=optimized@entry=false, osr_id=osr_id@entry=-1)

    at runtime/vm/compiler.cc:973

­9 0x081d7262 in CompileFunction (function=..., isolate=0xf68004a8) at runtime/vm/compiler.cc:1028

­10 DRT_HelperCompileFunction (isolate=0xf68004a8, arguments=...) at runtime/vm/compiler.cc:172

­11 dart::DRT_CompileFunction (arguments=...) at runtime/vm/compiler.cc:168

­12 0x083d3d26 in dart::Simulator::DoBreak (this=this@entry=0xf6817048, instr=instr@entry=0x8ee5704) at runtime/vm/simulator_mips.cc:1249

­13 0x083d494c in dart::Simulator::DecodeSpecial (this=this@entry=0xf6817048, instr=instr@entry=0x8ee5704) at runtime/vm/simulator_mips.cc:1365

­14 0x083d204c in dart::Simulator::InstructionDecode (this=0xf6817048, instr=0x8ee5704) at runtime/vm/simulator_mips.cc:1990

­15 0x083d60cc in dart::Simulator::Execute (this=this@entry=0xf6817048) at runtime/vm/simulator_mips.cc:2305

­16 0x083d638c in dart::Simulator::Call (this=0xf6817048, entry=entry@entry=-165413280, parameter0=parameter0@entry=-165300528,

    parameter1=parameter1@entry=-170916512, parameter2=parameter2@entry=-170916520, parameter3=parameter3@entry=0, fp_return=fp_return@entry=false,     fp_args=fp_args@entry=false) at runtime/vm/simulator_mips.cc:2407

­17 0x081eb3cf in dart::DartEntry::InvokeFunction (function=..., arguments=..., arguments_descriptor=...) at runtime/vm/dart_entry.cc:114

­18 0x081ec2e2 in dart::DartEntry::InvokeFunction (function=..., arguments=...) at runtime/vm/dart_entry.cc:27

­19 0x081ee6cf in dart::DartLibraryCalls::HandleMessage (handler=..., message=...) at runtime/vm/dart_entry.cc:499

­20 0x082dbf19 in dart::IsolateMessageHandler::HandleMessage (this=0xf6816648, message=0xf6100688) at runtime/vm/isolate.cc:370

­21 0x082e2b5a in dart::MessageHandler::HandleMessages (this=0xf6816648, allow_normal_messages=true, allow_multiple_normal_messages=true)

    at runtime/vm/message_handler.cc:160

­22 0x082e31d0 in TaskCallback (this=0xf6816648) at runtime/vm/message_handler.cc:246

­23 dart::MessageHandlerTask::Run (this=0xf61006b0) at runtime/vm/message_handler.cc:24

­24 0x083f494e in dart::ThreadPool::Worker::Loop (this=this@entry=0xf68ac768) at runtime/vm/thread_pool.cc:277

­25 0x083f4b23 in dart::ThreadPool::Worker::Main (args=4136290152) at runtime/vm/thread_pool.cc:318

­26 0x0834bc7d in dart::ThreadStart (data_ptr=0xf68ac7c8) at runtime/vm/os_thread_linux.cc:87

­27 0xf7746f70 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

­28 0xf752250e in clone () from /lib/i386-linux-gnu/libc.so.6

[kodandersson]

(gdb) t a a bt

Thread 5 (Thread 0xf6b73b40 (LWP 14785)):

­0 0xf7788430 in __kernel_vsyscall ()

­1 0xf774b12d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

­2 0x0834c912 in dart::Monitor::WaitMicros (this=this@entry=0x8ecaf58, micros=1000) at runtime/vm/os_thread_linux.cc:327

­3 0x083f39c0 in WaitMicros (this=<synthetic pointer>, micros=<optimized out>) at runtime/vm/lockers.h:54

­4 dart::ThreadInterrupter::ThreadMain (parameters=parameters@entry=0) at runtime/vm/thread_interrupter.cc:293

­5 0x0834bc7d in dart::ThreadStart (data_ptr=0x8ecac78) at runtime/vm/os_thread_linux.cc:87

­6 0xf7746f70 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

­7 0xf752250e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 4 (Thread 0xf69f1b40 (LWP 14803)):

­0 0xf7788430 in __kernel_vsyscall ()

­1 0xf774b12d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

­2 0x0834c912 in dart::Monitor::WaitMicros (this=0x8ee6b20, micros=5000000) at runtime/vm/os_thread_linux.cc:327

­3 0x0834c9a0 in dart::Monitor::Wait (this=this@entry=0x8ee6b20, millis=<optimized out>) at runtime/vm/os_thread_linux.cc:313

­4 0x083f49a7 in Wait (this=<synthetic pointer>, millis=<optimized out>) at runtime/vm/lockers.h:50

­5 dart::ThreadPool::Worker::Loop (this=this@entry=0x8ee6b20) at runtime/vm/thread_pool.cc:289

­6 0x083f4b23 in dart::ThreadPool::Worker::Main (args=149842720) at runtime/vm/thread_pool.cc:318

­7 0x0834bc7d in dart::ThreadStart (data_ptr=0x8ee6b80) at runtime/vm/os_thread_linux.cc:87

­8 0xf7746f70 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

­9 0xf752250e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 3 (Thread 0xf7432b40 (LWP 14784)):

­0 0xf7788430 in __kernel_vsyscall ()

­1 0xf7522fb6 in epoll_wait () from /lib/i386-linux-gnu/libc.so.6

­2 0x081b063b in dart::bin::EventHandlerImplementation::Poll (args=args@entry=149728432) at runtime/bin/eventhandler_linux.cc:319

­3 0x081aa73d in dart::bin::ThreadStart (data_ptr=0x8ecada8) at runtime/bin/thread_linux.cc:87

­4 0xf7746f70 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

­5 0xf752250e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 2 (Thread 0xf7434700 (LWP 14780)):

­0 Bits (count=5, shift=21, this=0xf6679ae0) at runtime/vm/constants_mips.h:527

­1 RsField (this=0xf6679ae0) at runtime/vm/constants_mips.h:542

­2 dart::Simulator::DecodeSpecial (this=this@entry=0x8ee4e90, instr=instr@entry=0xf6679ae0) at runtime/vm/simulator_mips.cc:1528

­3 0x083d204c in dart::Simulator::InstructionDecode (this=0x8ee4e90, instr=0xf6679ae0) at runtime/vm/simulator_mips.cc:1990

­4 0x083d60cc in dart::Simulator::Execute (this=this@entry=0x8ee4e90) at runtime/vm/simulator_mips.cc:2305

­5 0x083d638c in dart::Simulator::Call (this=0x8ee4e90, entry=entry@entry=-161210784, parameter0=parameter0@entry=-161107792,

    parameter1=parameter1@entry=150487004, parameter2=parameter2@entry=150486860, parameter3=parameter3@entry=0, fp_return=fp_return@entry=false,     fp_args=fp_args@entry=false) at runtime/vm/simulator_mips.cc:2407

­6 0x081eb3cf in dart::DartEntry::InvokeFunction (function=..., arguments=..., arguments_descriptor=...) at runtime/vm/dart_entry.cc:114

­7 0x081ec2e2 in dart::DartEntry::InvokeFunction (function=..., arguments=...) at runtime/vm/dart_entry.cc:27

---Type <return> to continue, or q <return> to quit---

­8 0x08188871 in dart::Dart_Invoke (target=0x8f83a28, name=0x8f83a70, number_of_arguments=1, arguments=0xffa78770) at runtime/vm/dart_api_impl.cc:3980

­9 0x08197b76 in dart::bin::DartUtils::PrepareBuiltinLibrary (builtin_lib=builtin_lib@entry=0x8f83a28, internal_lib=internal_lib@entry=0x8f83a48,

    is_service_isolate=is_service_isolate@entry=false, package_root=package_root@entry=0xffa795ab "out/ReleaseSIMMIPS/packages/")     at runtime/bin/dartutils.cc:659

­10 0x0819819e in dart::bin::DartUtils::PrepareForScriptLoading (package_root=package_root@entry=0xffa795ab "out/ReleaseSIMMIPS/packages/",

    is_service_isolate=is_service_isolate@entry=false, builtin_lib=builtin_lib@entry=0x8f83a28) at runtime/bin/dartutils.cc:734

­11 0x0816f9b0 in dart::bin::CreateIsolateAndSetupHelper (

    script_uri=script_uri@entry=0xffa795c8 "/usr/local/google/home/koda/flake/dart/tests/co19/src/Language/12_Expressions/24_Shift_A01_t13.dart",     main=main@entry=0x8a37a8a "main", package_root=0xffa795ab "out/ReleaseSIMMIPS/packages/", error=error@entry=0xffa78880,     is_compile_error=is_compile_error@entry=0xffa78870) at runtime/bin/main.cc:604

­12 0x08169c27 in dart::bin::main (argc=argc@entry=6, argv=argv@entry=0xffa789a4) at runtime/bin/main.cc:991

­13 0x0816a24b in main (argc=6, argv=0xffa789a4) at runtime/bin/main.cc:1124

Thread 1 (Thread 0xf653db40 (LWP 14826)):

­0 Get (argument_count=argument_count@entry=2, call_kind=dart::Simulator::kRuntimeCall, external_function=138550464) at runtime/vm/simulator_mips.cc:767

­1 dart::Simulator::RedirectExternalReference (function=138550464, call_kind=dart::Simulator::kRuntimeCall, argument_count=argument_count@entry=2)

    at runtime/vm/simulator_mips.cc:817

­2 0x0843e956 in dart::RuntimeEntry::Call (this=0x8bb1118 <dart::kUpdateFieldCidRuntimeEntry>, assembler=0xf653b370, argument_count=2)

    at runtime/vm/runtime_entry_mips.cc:41

­3 0x083ffa85 in dart::Assembler::CallRuntime (this=<optimized out>, entry=..., argument_count=argument_count@entry=2) at runtime/vm/assembler_mips.cc:971

­4 0x082ab9ad in dart::GuardFieldLengthInstr::EmitNativeCode (this=0xf5d12600, compiler=0xf653b470) at runtime/vm/intermediate_language_mips.cc:1821

­5 0x0823758c in dart::FlowGraphCompiler::VisitBlocks (this=this@entry=0xf653b470) at runtime/vm/flow_graph_compiler.cc:420

­6 0x0823c6c4 in dart::FlowGraphCompiler::CompileGraph (this=this@entry=0xf653b470) at runtime/vm/flow_graph_compiler_mips.cc:1144

­7 0x081d2210 in dart::CompileParsedFunctionHelper (pipeline=0xf653bf20, parsed_function=0xf653b850, optimized=false, osr_id=-1)

    at runtime/vm/compiler.cc:696

­8 0x081d3d33 in dart::CompileFunctionHelper (pipeline=0xf653bf20, function=..., optimized=optimized@entry=false, osr_id=osr_id@entry=-1)

    at runtime/vm/compiler.cc:973

­9 0x081d7262 in CompileFunction (function=..., isolate=0xf68004a8) at runtime/vm/compiler.cc:1028

­10 DRT_HelperCompileFunction (isolate=0xf68004a8, arguments=...) at runtime/vm/compiler.cc:172

­11 dart::DRT_CompileFunction (arguments=...) at runtime/vm/compiler.cc:168

­12 0x083d3d26 in dart::Simulator::DoBreak (this=this@entry=0xf6817048, instr=instr@entry=0x8ee5704) at runtime/vm/simulator_mips.cc:1249

­13 0x083d494c in dart::Simulator::DecodeSpecial (this=this@entry=0xf6817048, instr=instr@entry=0x8ee5704) at runtime/vm/simulator_mips.cc:1365

­14 0x083d204c in dart::Simulator::InstructionDecode (this=0xf6817048, instr=0x8ee5704) at runtime/vm/simulator_mips.cc:1990

­15 0x083d60cc in dart::Simulator::Execute (this=this@entry=0xf6817048) at runtime/vm/simulator_mips.cc:2305

­16 0x083d638c in dart::Simulator::Call (this=0xf6817048, entry=entry@entry=-165413280, parameter0=parameter0@entry=-165300528,

    parameter1=parameter1@entry=-170916512, parameter2=parameter2@entry=-170916520, parameter3=parameter3@entry=0, fp_return=fp_return@entry=false,     fp_args=fp_args@entry=false) at runtime/vm/simulator_mips.cc:2407

­17 0x081eb3cf in dart::DartEntry::InvokeFunction (function=..., arguments=..., arguments_descriptor=...) at runtime/vm/dart_entry.cc:114

­18 0x081ec2e2 in dart::DartEntry::InvokeFunction (function=..., arguments=...) at runtime/vm/dart_entry.cc:27

­19 0x081ee6cf in dart::DartLibraryCalls::HandleMessage (handler=..., message=...) at runtime/vm/dart_entry.cc:499

­20 0x082dbf19 in dart::IsolateMessageHandler::HandleMessage (this=0xf6816648, message=0xf6100688) at runtime/vm/isolate.cc:370

­21 0x082e2b5a in dart::MessageHandler::HandleMessages (this=0xf6816648, allow_normal_messages=true, allow_multiple_normal_messages=true)

---Type <return> to continue, or q <return> to quit---     at runtime/vm/message_handler.cc:160

­22 0x082e31d0 in TaskCallback (this=0xf6816648) at runtime/vm/message_handler.cc:246

­23 dart::MessageHandlerTask::Run (this=0xf61006b0) at runtime/vm/message_handler.cc:24

­24 0x083f494e in dart::ThreadPool::Worker::Loop (this=this@entry=0xf68ac768) at runtime/vm/thread_pool.cc:277

­25 0x083f4b23 in dart::ThreadPool::Worker::Main (args=4136290152) at runtime/vm/thread_pool.cc:318

­26 0x0834bc7d in dart::ThreadStart (data_ptr=0xf68ac7c8) at runtime/vm/os_thread_linux.cc:87

­27 0xf7746f70 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

­28 0xf752250e in clone () from /lib/i386-linux-gnu/libc.so.6

[kodandersson]

I now have several cases on DebugSIMARM/SIMMIPS where the Redirection linked list is corrupted, with a next_ pointer being either 0xabababab or 0x42424242, the pattern for uninitialized and deleted zone memory, respectively.

In all those cases, it's only the next_ pointer that is clobbered. And the pattern is surrounded by normal-looking values.

The Redirection class is unsynchronized and tries to do a lock-free linked list. Although there should not be any torn writes (we're on x86 and next_ is aligned), there might be other issues, such as reordering of stores by the compiler, which I will investigate.

BTW, in the days before the general flakiness was first observed on the build bots, there were several Zone-related changes which may be relevant: https://codereview.chromium.org/851513002/ https://codereview.chromium.org/832713006/ https://codereview.chromium.org/855533002/

[kodandersson]

The compiler reorders the stores, so the static head "list" is updated before the "next" pointer of the new element, as seen in the disassembly of the (inlined) constructor:

(gdb) print &list $19 = (dart::Redirection **) 0x8d40b60 <dart::Redirection::list> (gdb) set disassembly-flavor intel (gdb) disas dart::Simulator::RedirectExternalReference Dump of assembler code for function dart::Simulator::RedirectExternalReference(unsigned int, dart::Simulator::CallKind, int):  ...   0x084e8a81 <+81>: mov edx,DWORD PTR ds:0x8d40b60    0x084e8a87 <+87>: mov ds:0x8d40b60,eax    0x084e8a8c <+92>: add eax,0xc    0x084e8a8f <+95>: mov DWORD PTR [eax+0x4],edx ...

If the thread is interrupted between these two stores, it could lead to the corruption observed.

Since Redirection is only used on simulator builds, this raises the possibility that this class of crashes is separate from those seen on non-simulator builds.

[kodandersson]

The simulator failures should be fixed by https://codereview.chromium.org/898123002/

But that does not explain the non-simulator flaky crashes, so I'll keep this issue open to track those. Here's another recent one:

http://build.chromium.org/p/client.dart/builders/vm-win32-debug-be/builds/5571/steps/checked_tests/logs/stdio

... Stackmaps for function 'dart:convert_::_getLATIN1' { } Variable Descriptors for function 'dart:convert::_getLATIN1' {   saved current CTX reg offset -3 } Exception Handlers for function 'dart:convert::_get_LATIN1' { No exception handlers } Static call target functions { } Code for function 'dart:core_Uri__uriDecode@915557746' {

stderr: e:\b\build\slave\vm-win32-debug-be\build\dart\runtime\vm/class_table.h:132: error: expected: IsValidIndex(index)

Command[vm]: build\DebugIA32\dart.exe --disassemble --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=build/DebugIA32/packages/ e:\b\build\slave\vm-win32-debug-be\build\dart\tests\language\disassemble_test.dart Took 0:00:19.688000

Short reproduction command (experimental):     python tools/test.py --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t120 language/disassemble_test

=== === 1 test failed ===

---

Set owner to @kodandersson.

[kodandersson]

Another flaky crash, but this time in dart2js.

http://build.chromium.org/p/client.dart/builders/dart2js-linux-jsshell-release-2-4-be/builds/8047/steps/dart2js-jsshell%20tests%20--dart2js-batch%20failures/logs/stdio

@@@BUILD_STEP dart2js-jsshell tests --dart2js-batch failures@@@

FAILED: dart2js-jsshell release_ia32 lib/math/point_test Expected: Pass Actual: Crash Unexpected compile-time error. CommandOutput[dart2js]:

Command[dart2js]: out/ReleaseIA32/dart-sdk/bin/dart2js --allow-mock-compilation --categories=all --package-root=out/ReleaseIA32/packages/ /mnt/data/b/build/slave/dart2js-linux-jsshell-release-2-4-be/build/dart/tests/lib/math/point_test.dart --out=/mnt/data/b/build/slave/dart2js-linux-jsshell-release-2-4-be/build/dart/out/ReleaseIA32/generated_compilations/dart2js-sdk/tests_lib_math_point_test/out.js Took 0:00:04.761000

Command[jsshell]: /mnt/data/b/build/slave/dart2js-linux-jsshell-release-2-4-be/build/dart/tools/testing/bin/jsshell -f out/ReleaseIA32/dart-sdk/lib/_internal/compiler/js_lib/preambles/jsshell.js -f /mnt/data/b/build/slave/dart2js-linux-jsshell-release-2-4-be/build/dart/out/ReleaseIA32/generated_compilations/dart2js-sdk/tests_lib_math_point_test/out.js Did not run

Short reproduction command (experimental):     python tools/test.py -mrelease -cdart2js -rjsshell --use-sdk --write-debug-log --write-test-outcome-log --clear_browser_cache --dart2js-batch -t60 lib/math/point_test

=== === 1 test failed ===

[kodandersson]

With new, more detailed assertion failure:

http://build.chromium.org/p/client.dart/builders/vm-arm-sim-debug-be/builds/3233/steps/checked_tests/logs/stdio

FAILED: none-vm-checked debug_simarm lib/convert/streamed_conversion_json_utf8_decode_test Expected: Pass Actual: Crash CommandOutput[vm]:

stderr: Verifying before marking... done. Verifying before sweeping...Verifying before marking... done.  done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. Verifying before marking... done. Verifying before sweeping... done. runtime/vm/raw_object.cc:221: error: Size mismatch: 32 from class vs 24 from tags 3e0300

Command[vm]: DART_CONFIGURATION=DebugSIMARM out/DebugSIMARM/dart --verified_mem --verify_before_gc --verify_after_gc --old_gen_growth_rate=1 --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=out/DebugSIMARM/packages/ /mnt/data/b/build/slave/vm-arm-sim-debug-be/build/dart/tests/lib/convert/streamed_conversion_json_utf8_decode_test.dart Took 0:00:01.457000

Short reproduction command (experimental):     python tools/test.py -asimarm --write-debug-log --write-test-outcome-log --copy-coredumps --exclude-suite pkg --checked -t480 lib/convert/streamed_conversion_json_utf8_decode_test

=== === 1 test failed ===

[kodandersson]

Size assertion failures are probably caused by a debug-only race. Fix/workaround is under review here: https://codereview.chromium.org/936393003/

[kodandersson]

Closing this as too broad.

Multiple independent crash issues have been fixed, although there are still occasional crashes, also in release mode (in particular, also the VM driving the test harness).

If/when we have any details/pattern, and ideally a core dump/repro, we should file a new issue.

---

Added TooBroad label.