search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.53k stars 435 forks source link

Symbolic name registration causes application hangs #1034

Closed brycelelbach closed 10 years ago

brycelelbach commented 10 years ago

Running Fibonacci on ariel00 with the below options and the below git hash leads to a hang:

[23:12:11]:wash@ariel00.hermione:/.../gcc-4.6.2-release:0:$ ~/install/hpx/gcc-4.6.2-release/bin/fibonacci \
    --hpx:print-counter /threads{locality#*/worker-thread#*}/idle-rate \
    --hpx:print-counter-interval 1 -t16 --n-value 20 --version

HPX - High Performance ParalleX
A general purpose parallel C++ runtime system for distributed applications
of any scale.

Copyright (c) 2007-2013 The STE||AR Group, Louisiana State University,
http://stellar.cct.lsu.edu, email:hpx-users@stellar.cct.lsu.edu

Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

Versions:
  HPX: V0.9.8-trunk (AGAS: V3.0), Git: 4fada5472f31d87b25618ca6f62f993ebc99d7b4
  Boost: V1.53.0
  Hwloc: V1.7.0

Build:
  Type: release
  Date: Dec 17 2013 22:39:44
  Platform: linux
  Compiler: GNU C++ version 4.6.4
  Standard Library: GNU libstdc++ version 20130412

A stack trace from all threads reveals that all the scheduler threads are looking for work, except for one thread, which sits in addressing_space::register_name_async attempting to acquire a spinlock. It consistently remains in spinlock acquisition.

Thread 24 (Thread 0x2aaab1262700 (LWP 2141)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1262700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 23 (Thread 0x2aaab1463700 (LWP 2142)):
#0  0x00002aaaae0f2f23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabb8b64b in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabd56324 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1463700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 22 (Thread 0x2aaab1664700 (LWP 2143)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1664700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 21 (Thread 0x2aaab1865700 (LWP 2144)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1865700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 20 (Thread 0x2aaab1a66700 (LWP 2145)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1a66700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 19 (Thread 0x2aaab1c67700 (LWP 2146)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1c67700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 18 (Thread 0x2aaab1e68700 (LWP 2147)):
#0  __pthread_mutex_unlock_usercnt (mutex=0x2aaaaaef9200, decr=1) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1e68700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 17 (Thread 0x2aaab2069700 (LWP 2148)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef7900) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2069700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 16 (Thread 0x2aaab226a700 (LWP 2149)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaab0fea780) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1842c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab226a700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 15 (Thread 0x2aaab246b700 (LWP 2150)):
#0  0x00002aaaabc17f67 in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab246b700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 14 (Thread 0x2aaab266c700 (LWP 2151)):
#0  0x00002aaaabc181ca in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab266c700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 13 (Thread 0x2aaab286d700 (LWP 2152)):
#0  __GI___pthread_mutex_unlock (mutex=0x2aaaaaef8300) at pthread_mutex_unlock.c:297
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1814c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab286d700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 12 (Thread 0x2aaab2a6e700 (LWP 2153)):
#0  __pthread_mutex_unlock_usercnt (mutex=0x2aaab0feb180, decr=1) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1842c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2a6e700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 11 (Thread 0x2aaab2c6f700 (LWP 2154)):
#0  0x00002aaaab8c2550 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2c6f700) at pthread_create.c:311
#6  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 10 (Thread 0x2aaab2e70700 (LWP 2155)):
#0  0x00002aaaab91d3c0 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#6  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2e70700) at pthread_create.c:311
#7  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 9 (Thread 0x2aaab3071700 (LWP 2156)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef7680) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3071700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x2aaab3272700 (LWP 2157)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef6780) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3272700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x2aaab3473700 (LWP 2158)):
#0  0x00002aaaab927e66 in hpx::threads::policies::thread_queue<boost::mutex>::cleanup_terminated_locked_helper(bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaab8c2606 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#6  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3473700) at pthread_create.c:311
#7  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x2aaab3674700 (LWP 2159)):
#0  0x00002aaaaddfcacd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaaba2e4e3 in hpx::util::spinlock_pool<hpx::naming::gid_type::tag>::scoped_lock::lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaba5ce2d in hpx::lcos::future<bool> hpx::agas::stubs::symbol_namespace::service_async<bool>(hpx::naming::id_type const&, hpx::agas::request const&, hpx::threads::thread_priority) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaba90047 in hpx::agas::addressing_service::register_name_async(std::string const&, hpx::naming::id_type const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaba76e77 in hpx::agas::register_name_sync(std::string const&, hpx::naming::id_type const&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabd13e5c in hpx::performance_counters::register_with_agas(hpx::lcos::future<hpx::naming::id_type>, std::string const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabd1f594 in void hpx::lcos::detail::invoke_continuation<hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::lcos::future<hpx::naming::id_type>, hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type> >(hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >&, hpx::lcos::future<hpx::naming::id_type>&, hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type>&, mpl_::bool_<false>) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabd1fbf3 in hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type>::run(boost::intrusive_ptr<hpx::lcos::detail::future_data<hpx::naming::id_type> > const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaab99fee8 in void hpx::lcos::detail::future_data<hpx::naming::id_type>::set_result<hpx::naming::id_type>(hpx::naming::id_type&&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#9  0x00002aaaab9a00a4 in hpx::lcos::detail::promise<hpx::naming::id_type, hpx::naming::gid_type>::set_value(hpx::naming::gid_type&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#10 0x00002aaaab99e605 in void hpx::applier::detail::apply_helper<hpx::actions::direct_action1<void (hpx::lcos::base_lco_with_value<hpx::naming::gid_type, hpx::naming::gid_type>::*)(hpx::naming::gid_type&&), &hpx::lcos::base_lco_with_value<hpx::naming::gid_type, hpx::naming::gid_type>::set_value_nonvirt, hpx::actions::detail::this_type>, mpl_::bool_<true> >::call<hpx::util::tuple<hpx::naming::gid_type&&, void, void, void, void, void, void, void> >(hpx::naming::id_type const&, unsigned long, hpx::threads::thread_priority, hpx::util::tuple<hpx::naming::gid_type&&, void, void, void, void, void, void, void>&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#11 0x00002aaaab99ead8 in hpx::actions::typed_continuation<hpx::naming::gid_type>::trigger_value(hpx::naming::gid_type&&) const () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#12 0x00002aaaabb285dc in hpx::util::detail::vtable<false>::type<hpx::util::detail::bound<hpx::actions::action<hpx::components::server::runtime_support, hpx::naming::gid_type, hpx::util::tuple<hpx::performance_counters::counter_info, void, void, void, void, void, void, void>, hpx::actions::result_action1<hpx::naming::gid_type (hpx::components::server::runtime_support::*)(hpx::performance_counters::counter_info const&), &hpx::components::server::runtime_support::create_performance_counter, hpx::actions::detail::this_type> >::continuation_thread_object_function_1, hpx::util::tuple<boost::shared_ptr<hpx::actions::continuation>, hpx::naming::gid_type (hpx::components::server::runtime_support::*)(hpx::performance_counters::counter_info const&), hpx::components::server::runtime_support*, hpx::performance_counters::counter_info, void, void, void, void> >, hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), void, void>::invoke(void**, hpx::threads::thread_state_ex_enum&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#13 0x00002aaaab959469 in hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>::operator()() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#14 0x00002aaaab90f589 in void hpx::util::coroutines::detail::lx::trampoline<hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#15 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x2aaab3875700 (LWP 2160)):
#0  __pthread_mutex_trylock (mutex=0x2aaaaaef6280) at pthread_mutex_trylock.c:29
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3875700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x2aaab3a76700 (LWP 2161)):
#0  0x00002aaaaddf8e1f in __pthread_mutex_unlock_usercnt (mutex=0x2aaaaaef6500, decr=<optimized out>) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3a76700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x2aaab3c77700 (LWP 2162)):
#0  0x00002aaaabc18251 in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3c77700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x2aaab417a700 (LWP 2163)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabb13185 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) [clone .constprop.3950] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabb171ea in hpx::components::server::runtime_support::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaab91d75f in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait_helper(boost::mutex&, boost::condition_variable_any&, bool&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab417a700) at pthread_create.c:311
#6  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x2aaaaad5b480 (LWP 2131)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab93ac20 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaab914755 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run(hpx::util::function_nonser<int ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaab967cec in hpx::detail::run(hpx::runtime&, hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::variables_map&, hpx::runtime_mode, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaab968d70 in hpx::detail::run_priority_local(hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::util::command_line_handling&, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaab96ae6b in hpx::run_or_start(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaab96b3a2 in hpx::init(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaaab12619 in main ()

The problem does not show up without the performance counter.

brycelelbach commented 10 years ago

Occassionally, if I use a performance counter with an interval, I get a similar hang at application shutdown.

brycelelbach commented 10 years ago

This actually affects all applications as the race condition is occasionally hit for the bootstrap barriers:

Thread 9 (Thread 0x2aaab12f4700 (LWP 4720)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab12f4700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x2aaab14f5700 (LWP 4721)):
#0  0x00002aaaae285f23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabd1e64b in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabee9324 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaadf88e0e in start_thread (arg=0x2aaab14f5700) at pthread_create.c:311
#5  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x2aaab16f6700 (LWP 4722)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab16f6700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x2aaab18f7700 (LWP 4723)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab18f7700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x2aaab1af8700 (LWP 4724)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab1af8700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x2aaab1cf9700 (LWP 4725)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab1cf9700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x2aaab1efa700 (LWP 4726)):
#0  0x00002aaaadf8facd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabbc14e3 in hpx::util::spinlock_pool<hpx::naming::gid_type::tag>::scoped_lock::lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabbefe2d in hpx::lcos::future<bool> hpx::agas::stubs::symbol_namespace::service_async<bool>(hpx::naming::id_type const&, hpx::agas::request const&, hpx::threads::thread_priority) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc23047 in hpx::agas::addressing_service::register_name_async(std::string const&, hpx::naming::id_type const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc09e77 in hpx::agas::register_name_sync(std::string const&, hpx::naming::id_type const&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabb2e94b in hpx::create_barrier(unsigned long, char const*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabb21068 in hpx::pre_main(hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabacef22 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run_helper(hpx::util::function_nonser<int ()>, int&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaabaa3343 in hpx::util::detail::vtable<false>::type<boost::_bi::bind_t<hpx::threads::detail::tagged_thread_state<hpx::threads::thread_state_enum>, boost::_mfi::mf2<hpx::threads::detail::tagged_thread_state<hpx::threads::thread_state_enum>, hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, hpx::util::function_nonser<int ()>, int&>, boost::_bi::list3<boost::_bi::value<hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<hpx::util::function_nonser<int ()> >, boost::reference_wrapper<int> > >, hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), void, void>::invoke(void**, hpx::threads::thread_state_ex_enum&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#9  0x00002aaaabaec469 in hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>::operator()() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#10 0x00002aaaabaa2589 in void hpx::util::coroutines::detail::lx::trampoline<hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#11 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x2aaab22fc700 (LWP 4727)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabca6185 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) [clone .constprop.3950] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabcaa1ea in hpx::components::server::runtime_support::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabab075f in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait_helper(boost::mutex&, boost::condition_variable_any&, bool&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaadf88e0e in start_thread (arg=0x2aaab22fc700) at pthread_create.c:311
#6  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x2aaaaacf0b40 (LWP 4715)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabacdc20 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabaa7755 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run(hpx::util::function_nonser<int ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabafacec in hpx::detail::run(hpx::runtime&, hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::variables_map&, hpx::runtime_mode, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabafbd70 in hpx::detail::run_priority_local(hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::util::command_line_handling&, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabafde6b in hpx::run_or_start(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabafe3a2 in hpx::init(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaaaad5d9f in main ()
hkaiser commented 10 years ago

Please enable lock tracking and report back whether this triggers some held lock during suspension.

sithhell commented 10 years ago

Please try again with the latest commit. I am still seeing occasional hangs on shutdown. Investigating.

hkaiser commented 10 years ago

What's the current status of this?

sithhell commented 10 years ago

I was not able to reproduce this specific hang. Just recently ran a job on 256 cores with around 1024 performance counter enabled. Please reopen if the problem is still there.