search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.21k stars 1.57k forks source link

lib_2/isolate/spawn_uri_nested_vm_test sporadically hangs in app_jitk configuration #33385

Open alexmarkov opened 6 years ago

alexmarkov commented 6 years ago 
tools/test.py -m release -c app_jitk --strong lib_2/isolate/spawn_uri_nested_vm_test

Easily reproducible on Linux - hanged 4 out of 5 runs. This test spawns a new isolate which spawns a yet another isolate. It looks like a newly spawned isolate is waiting for kernel service isolate, but it has gone already (as main isolate is shutting down and waiting for other isolates to finish).

Maybe we should not kill kernel service isolate during shutdown until all application isolates are terminated (similarly to VM service isolate).

Stack traces:

Thread 10 (Thread 0x7ffff7fb7700 (LWP 248384)):
#0  0x00007ffff6c59083 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00000000006e3885 in dart::bin::EventHandlerImplementation::Poll (args=39370816) at ../../runtime/bin/eventhandler_linux.cc:389
#2  0x00000000007023be in dart::bin::ThreadStart (data_ptr=<optimized out>) at ../../runtime/bin/thread_linux.cc:85
#3  0x00007ffff79bf494 in start_thread (arg=0x7ffff7fb7700) at pthread_create.c:333
#4  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7ffff663f700 (LWP 248385)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000009fbcd8 in dart::Monitor::WaitMicros (this=0x286e420, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:444
#2  0x0000000000952e6a in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::KernelCompilationRequest::SendAndWaitForResponse (this=0x7ffff663e9c8, request_tag=<optimized out>, kernel_port=976596681, 
    script_uri=<optimized out>, platform_kernel=<optimized out>, platform_kernel_size=5512147, source_files_count=0, source_files=0x0, 
    incremental_compile=<optimized out>, package_config=0x31a2f78 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages")
    at ../../runtime/vm/kernel_isolate.cc:581
#4  0x0000000000952a1f in dart::KernelIsolate::CompileToKernel (
    script_uri=0x2596580 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/tests/lib_2/isolate/spawn_uri_nested_child2_vm_isolate.dart", 
    platform_kernel=0x1660a80 <kPlatformStrongDill> "\220\253\315", <incomplete sequence \357>, platform_kernel_size=5512147, source_file_count=0, 
    source_files=0x0, incremental_compile=<optimized out>, package_config=0x31a2f78 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages")
    at ../../runtime/vm/kernel_isolate.cc:713
#5  0x0000000000c76157 in Dart_CompileToKernel (script_uri=0x80 <error: Cannot access memory at address 0x80>, 
    platform_kernel=0x1 <error: Cannot access memory at address 0x1>, platform_kernel_size=140737347604831, incremental_compile=false, 
    package_config=0xfffffffffffffe00 <error: Cannot access memory at address 0xfffffffffffffe00>) at ../../runtime/vm/dart_api_impl.cc:5910
#6  0x00000000006e0045 in CompileScript (script_uri=0x80 <error: Cannot access memory at address 0x80>, incremental=255, 
    package_config=0x31a2f78 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages", this=<optimized out>, strong=<optimized out>)
    at ../../runtime/bin/dfe.cc:207
#7  dart::bin::DFE::CompileAndReadScript (this=<optimized out>, script_uri=0x80 <error: Cannot access memory at address 0x80>, kernel_buffer=0x7ffff663eb50, 
    kernel_buffer_size=0x7ffff663eb20, error=0x7ffff663ed50, exit_code=0x7ffff663ecb4, strong=<optimized out>, 
    package_config=0x31a2f78 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages") at ../../runtime/bin/dfe.cc:220
#8  0x00000000006dc139 in dart::bin::IsolateSetupHelper (isolate=0x2d35b00, is_main_isolate=false, 
    script_uri=0x2596580 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/tests/lib_2/isolate/spawn_uri_nested_child2_vm_isolate.dart", 
    package_root=<optimized out>, packages_config=<optimized out>, set_native_resolvers=true, isolate_run_app_snapshot=false, flags=0x7ffff663ed00, 
    error=0x7ffff663ed50, exit_code=0x7ffff663ecb4) at ../../runtime/bin/main.cc:331
#9  0x00000000006daf68 in dart::bin::CreateIsolateAndSetupHelper (is_main_isolate=<optimized out>, 
    script_uri=0x2596580 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/tests/lib_2/isolate/spawn_uri_nested_child2_vm_isolate.dart", 
    main=<optimized out>, package_root=0x0, packages_config=0x2586050 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages", 
    flags=0x7ffff663ed00, error=0x7ffff663ed50, exit_code=0x7ffff663ecb4) at ../../runtime/bin/main.cc:710
#10 0x00000000006db95e in dart::bin::CreateIsolateAndSetup (
    script_uri=0x2596580 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/tests/lib_2/isolate/spawn_uri_nested_child2_vm_isolate.dart", 
    main=0x257a030 "main", package_root=0x0, package_config=0x2586050 "file:///usr/local/google/home/alexmarkov/work/dart/sdk/.packages", 
    flags=0x7ffff663ed00, data=<optimized out>, error=0x7ffff663ed50) at ../../runtime/bin/main.cc:753
#11 0x000000000087ea84 in dart::SpawnIsolateTask::Run (this=0x257c550) at ../../runtime/lib/isolate.cc:132
#12 0x0000000000ae442d in dart::ThreadPool::Worker::Loop (this=0x2584090) at ../../runtime/vm/thread_pool.cc:381
#13 0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338128) at ../../runtime/vm/thread_pool.cc:436
#14 0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#15 0x00007ffff79bf494 in start_thread (arg=0x7ffff663f700) at pthread_create.c:333
#16 0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7ffff653e700 (LWP 248386)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000009fbcc0 in dart::Monitor::WaitMicros (this=0x2584120, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:449
#2  0x0000000000ae44ee in WaitMicros (micros=5000000, this=<optimized out>) at ../../runtime/vm/lockers.h:177
#3  dart::ThreadPool::Worker::Loop (this=0x2584120) at ../../runtime/vm/thread_pool.cc:394
#4  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338272) at ../../runtime/vm/thread_pool.cc:436
#5  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#6  0x00007ffff79bf494 in start_thread (arg=0x7ffff653e700) at pthread_create.c:333
#7  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7ffff09bf700 (LWP 248387)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000009fbcc0 in dart::Monitor::WaitMicros (this=0x25841b0, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:449
#2  0x0000000000ae44ee in WaitMicros (micros=4999999, this=<optimized out>) at ../../runtime/vm/lockers.h:177
#3  dart::ThreadPool::Worker::Loop (this=0x25841b0) at ../../runtime/vm/thread_pool.cc:394
#4  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338416) at ../../runtime/vm/thread_pool.cc:436
#5  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#6  0x00007ffff79bf494 in start_thread (arg=0x7ffff09bf700) at pthread_create.c:333
#7  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7ffff013f700 (LWP 248388)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000009fbcc0 in dart::Monitor::WaitMicros (this=0x2584240, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:449
#2  0x0000000000ae44ee in WaitMicros (micros=5000000, this=<optimized out>) at ../../runtime/vm/lockers.h:177
#3  dart::ThreadPool::Worker::Loop (this=0x2584240) at ../../runtime/vm/thread_pool.cc:394
#4  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338560) at ../../runtime/vm/thread_pool.cc:436
#5  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#6  0x00007ffff79bf494 in start_thread (arg=0x7ffff013f700) at pthread_create.c:333
#7  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7fffefd3f700 (LWP 248389)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000009fbcc0 in dart::Monitor::WaitMicros (this=0x25842d0, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:449
#2  0x0000000000ae44ee in WaitMicros (micros=5000000, this=<optimized out>) at ../../runtime/vm/lockers.h:177
#3  dart::ThreadPool::Worker::Loop (this=0x25842d0) at ../../runtime/vm/thread_pool.cc:394
#4  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338704) at ../../runtime/vm/thread_pool.cc:436
#5  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#6  0x00007ffff79bf494 in start_thread (arg=0x7fffefd3f700) at pthread_create.c:333
#7  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7fffed17f700 (LWP 248390)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000009fbcd8 in dart::Monitor::WaitMicros (this=0x2577a40, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:444
#2  0x000000000093bdfa in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  WaitForOutstandingSpawns (this=0x2d34900) at ../../runtime/vm/isolate.cc:2697
#4  dart::ShutdownIsolate (parameter=47401216) at ../../runtime/vm/isolate.cc:1602
#5  0x0000000000962968 in dart::MessageHandler::TaskCallback (this=0x25c0b60) at ../../runtime/vm/message_handler.cc:496
#6  0x0000000000ae442d in dart::ThreadPool::Worker::Loop (this=0x2584360) at ../../runtime/vm/thread_pool.cc:381
#7  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338848) at ../../runtime/vm/thread_pool.cc:436
#8  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#9  0x00007ffff79bf494 in start_thread (arg=0x7fffed17f700) at pthread_create.c:333
#10 0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fffecfff700 (LWP 248391)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000009fbcd8 in dart::Monitor::WaitMicros (this=0x2576900, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:444
#2  0x0000000000c330aa in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  dart::BackgroundCompiler::Run (this=0x25752c0) at ../../runtime/vm/compiler/jit/compiler.cc:1788
#4  0x0000000000ae442d in dart::ThreadPool::Worker::Loop (this=0x25843f0) at ../../runtime/vm/thread_pool.cc:381
#5  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39338992) at ../../runtime/vm/thread_pool.cc:436
#6  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#7  0x00007ffff79bf494 in start_thread (arg=0x7fffecfff700) at pthread_create.c:333
#8  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fffe2c7f700 (LWP 248416)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000009fbcc0 in dart::Monitor::WaitMicros (this=0x25845a0, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:449
#2  0x0000000000ae44ee in WaitMicros (micros=5000000, this=<optimized out>) at ../../runtime/vm/lockers.h:177
#3  dart::ThreadPool::Worker::Loop (this=0x25845a0) at ../../runtime/vm/thread_pool.cc:394
#4  0x0000000000ae42d1 in dart::ThreadPool::Worker::Main (args=39339424) at ../../runtime/vm/thread_pool.cc:436
#5  0x00000000009fb479 in dart::ThreadStart (data_ptr=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:135
#6  0x00007ffff79bf494 in start_thread (arg=0x7fffe2c7f700) at pthread_create.c:333
#7  0x00007ffff6c58a8f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7ffff7fbac00 (LWP 248380)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000009fbcd8 in dart::Monitor::WaitMicros (this=0x25762a0, micros=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:444
#2  0x00000000008fddba in Wait (millis=0, this=<optimized out>) at ../../runtime/vm/lockers.h:169
#3  WaitForApplicationIsolateShutdown () at ../../runtime/vm/dart.cc:339
#4  dart::Dart::Cleanup () at ../../runtime/vm/dart.cc:414
#5  0x0000000000c51c69 in Dart_Cleanup () at ../../runtime/vm/dart_api_impl.cc:1007
#6  0x00000000006db6c7 in dart::bin::main (argc=9, argv=0x7fffffffdad8) at ../../runtime/bin/main.cc:1327
#7  0x00000000006dbef9 in main (argc=39281356, argv=0x80) at ../../runtime/bin/main.cc:1358

/cc @rmacnak-google @a-siva

rmacnak-google commented 6 years ago

Agree, the lifecycle of the kernel isolate should be really similar to the lifecycle of the service isolate. I'm surprised it isn't already because I'd figure that would have been the easier thing to do when it was first added.

srawlins commented 5 years ago

Related: #28192.