search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.51k stars 428 forks source link

Race condition at shutdown #894

Closed brycelelbach closed 10 years ago

brycelelbach commented 10 years ago

Sameer initial ran into this. When using perf counters with intervals, occasionally applications hang at shutdown. Sameer ran into this specifically with jacobi. If you remove the counter interval, the problem goes away.

While trying to reproduce this, I ran into:

[11:36:34]:wash@ariel00.hermione:/srv/scratch/wash/hpx/gcc-4.6.2-debug:0:$ ~/install/hpx/gcc-4.6.2-debug/bin/jacobi --hpx:print-counter /threadqueue{locality#*/total}/length --hpx:print-counter-interval 10 -t16 --nx 100 --ny 100 --hpx:debug-hpx-log='/tmp/foobar'
/threadqueue{locality#0/total}/length,1,0.057813,[s],0
/threadqueue{locality#0/total}/length,2,0.069163,[s],0
/threadqueue{locality#0/total}/length,3,0.080520,[s],0
/threadqueue{locality#0/total}/length,4,0.092200,[s],0
/threadqueue{locality#0/total}/length,5,0.103634,[s],0
/threadqueue{locality#0/total}/length,6,0.116258,[s],0
/threadqueue{locality#0/total}/length,7,0.127839,[s],4
/threadqueue{locality#0/total}/length,8,0.140702,[s],6
/threadqueue{locality#0/total}/length,9,0.153713,[s],0
/threadqueue{locality#0/total}/length,10,0.166569,[s],15
/threadqueue{locality#0/total}/length,11,0.183908,[s],188
/threadqueue{locality#0/total}/length,12,0.203572,[s],296
/threadqueue{locality#0/total}/length,13,0.219059,[s],467
/threadqueue{locality#0/total}/length,14,0.239922,[s],805
/threadqueue{locality#0/total}/length,15,0.254971,[s],985
/threadqueue{locality#0/total}/length,16,0.274396,[s],1075
/threadqueue{locality#0/total}/length,17,0.316999,[s],1195
/threadqueue{locality#0/total}/length,18,0.336190,[s],1287
/threadqueue{locality#0/total}/length,19,0.361825,[s],1134
/threadqueue{locality#0/total}/length,20,0.375448,[s],1144
/threadqueue{locality#0/total}/length,21,0.388000,[s],1128
/threadqueue{locality#0/total}/length,22,0.401783,[s],1141
/threadqueue{locality#0/total}/length,23,0.415038,[s],1130
/threadqueue{locality#0/total}/length,24,0.433257,[s],1080
/threadqueue{locality#0/total}/length,25,0.452430,[s],864
/threadqueue{locality#0/total}/length,26,0.470491,[s],711
/threadqueue{locality#0/total}/length,27,0.483367,[s],715
/threadqueue{locality#0/total}/length,28,0.495941,[s],585
/threadqueue{locality#0/total}/length,29,0.508105,[s],405
/threadqueue{locality#0/total}/length,30,0.520677,[s],271
/threadqueue{locality#0/total}/length,31,0.537564,[s],28
/threadqueue{locality#0/total}/length,32,0.549851,[s],28
/threadqueue{locality#0/total}/length,33,0.567388,[s],209
/threadqueue{locality#0/total}/length,34,0.599767,[s],566
/threadqueue{locality#0/total}/length,35,0.619755,[s],671
/threadqueue{locality#0/total}/length,36,0.650438,[s],817
/threadqueue{locality#0/total}/length,37,0.669644,[s],805
/threadqueue{locality#0/total}/length,38,0.682192,[s],844
/threadqueue{locality#0/total}/length,39,0.708019,[s],828
/threadqueue{locality#0/total}/length,40,0.720625,[s],777
/threadqueue{locality#0/total}/length,41,0.733635,[s],756
/threadqueue{locality#0/total}/length,42,0.749302,[s],811
/threadqueue{locality#0/total}/length,43,0.764035,[s],843
/threadqueue{locality#0/total}/length,44,0.777331,[s],788
/threadqueue{locality#0/total}/length,45,0.789402,[s],672
/threadqueue{locality#0/total}/length,46,0.802306,[s],611
/threadqueue{locality#0/total}/length,47,0.814833,[s],510
/threadqueue{locality#0/total}/length,48,0.843142,[s],483
/threadqueue{locality#0/total}/length,49,0.886697,[s],263
/threadqueue{locality#0/total}/length,50,0.900186,[s],115
/threadqueue{locality#0/total}/length,51,0.912219,[s],8
/threadqueue{locality#0/total}/length,52,0.927537,[s],30
/threadqueue{locality#0/total}/length,53,0.940744,[s],185
/threadqueue{locality#0/total}/length,54,0.959639,[s],441
/threadqueue{locality#0/total}/length,55,0.980662,[s],680
/threadqueue{locality#0/total}/length,56,0.995418,[s],673
/threadqueue{locality#0/total}/length,57,1.025120,[s],684
/threadqueue{locality#0/total}/length,58,1.043148,[s],754
/threadqueue{locality#0/total}/length,59,1.070723,[s],845
/threadqueue{locality#0/total}/length,60,1.083731,[s],910
/threadqueue{locality#0/total}/length,61,1.106773,[s],916
/threadqueue{locality#0/total}/length,62,1.120259,[s],837
/threadqueue{locality#0/total}/length,63,1.132974,[s],886
/threadqueue{locality#0/total}/length,64,1.151268,[s],913
/threadqueue{locality#0/total}/length,65,1.167832,[s],888
/threadqueue{locality#0/total}/length,66,1.180117,[s],932
/threadqueue{locality#0/total}/length,67,1.192266,[s],938
/threadqueue{locality#0/total}/length,68,1.204830,[s],834
/threadqueue{locality#0/total}/length,69,1.217442,[s],700
/threadqueue{locality#0/total}/length,70,1.230078,[s],532
/threadqueue{locality#0/total}/length,71,1.242386,[s],402
/threadqueue{locality#0/total}/length,72,1.254764,[s],303
/threadqueue{locality#0/total}/length,73,1.268161,[s],213
/threadqueue{locality#0/total}/length,74,1.280104,[s],89
/threadqueue{locality#0/total}/length,75,1.292090,[s],0
/threadqueue{locality#0/total}/length,76,1.304745,[s],21
/threadqueue{locality#0/total}/length,77,1.329637,[s],362
/threadqueue{locality#0/total}/length,78,1.346404,[s],617
/threadqueue{locality#0/total}/length,79,1.524646,[s],1565
/threadqueue{locality#0/total}/length,80,1.537759,[s],1531
/threadqueue{locality#0/total}/length,81,1.549898,[s],1420
/threadqueue{locality#0/total}/length,82,1.563930,[s],1240
/threadqueue{locality#0/total}/length,83,1.576507,[s],1082
/threadqueue{locality#0/total}/length,84,1.589651,[s],1025
/threadqueue{locality#0/total}/length,85,1.602611,[s],973
/threadqueue{locality#0/total}/length,86,1.615394,[s],850
/threadqueue{locality#0/total}/length,87,1.627921,[s],638
/threadqueue{locality#0/total}/length,88,1.639861,[s],462
/threadqueue{locality#0/total}/length,89,1.651865,[s],259
/threadqueue{locality#0/total}/length,90,1.663977,[s],157
/threadqueue{locality#0/total}/length,91,1.676160,[s],15
/threadqueue{locality#0/total}/length,92,1.688676,[s],30
/threadqueue{locality#0/total}/length,93,1.706919,[s],267
/threadqueue{locality#0/total}/length,94,1.718367,[s],427
/threadqueue{locality#0/total}/length,95,1.733932,[s],519
/threadqueue{locality#0/total}/length,96,1.763881,[s],795
/threadqueue{locality#0/total}/length,97,1.794631,[s],1119
/threadqueue{locality#0/total}/length,98,1.852072,[s],1341
/threadqueue{locality#0/total}/length,99,1.868972,[s],1410
/threadqueue{locality#0/total}/length,100,1.888228,[s],1375
/threadqueue{locality#0/total}/length,101,1.926765,[s],1443
/threadqueue{locality#0/total}/length,102,1.939151,[s],1407
/threadqueue{locality#0/total}/length,103,1.951660,[s],1348
/threadqueue{locality#0/total}/length,104,1.964307,[s],1310
/threadqueue{locality#0/total}/length,105,1.977000,[s],1155
/threadqueue{locality#0/total}/length,106,1.992746,[s],902
/threadqueue{locality#0/total}/length,107,2.004902,[s],692
/threadqueue{locality#0/total}/length,108,2.018192,[s],504
/threadqueue{locality#0/total}/length,109,2.030778,[s],337
/threadqueue{locality#0/total}/length,110,2.043093,[s],157
/threadqueue{locality#0/total}/length,111,2.055112,[s],10
/threadqueue{locality#0/total}/length,112,2.066945,[s],47
/threadqueue{locality#0/total}/length,113,2.092539,[s],319
/threadqueue{locality#0/total}/length,114,2.113392,[s],572
/threadqueue{locality#0/total}/length,115,2.148198,[s],519
/threadqueue{locality#0/total}/length,116,2.196307,[s],662
/threadqueue{locality#0/total}/length,117,2.224641,[s],958
/threadqueue{locality#0/total}/length,118,2.237482,[s],1010
/threadqueue{locality#0/total}/length,119,2.253141,[s],976
/threadqueue{locality#0/total}/length,120,2.265534,[s],919
/threadqueue{locality#0/total}/length,121,2.277865,[s],920
/threadqueue{locality#0/total}/length,122,2.292616,[s],879
/threadqueue{locality#0/total}/length,123,2.307451,[s],846
/threadqueue{locality#0/total}/length,124,2.319521,[s],840
/threadqueue{locality#0/total}/length,125,2.333801,[s],684
/threadqueue{locality#0/total}/length,126,2.347149,[s],611
/threadqueue{locality#0/total}/length,127,2.365178,[s],576
/threadqueue{locality#0/total}/length,128,2.422996,[s],188
/threadqueue{locality#0/total}/length,129,2.436127,[s],21
/threadqueue{locality#0/total}/length,130,2.456239,[s],117
/threadqueue{locality#0/total}/length,131,2.473063,[s],236
/threadqueue{locality#0/total}/length,132,2.490865,[s],430
/threadqueue{locality#0/total}/length,133,2.512201,[s],554
/threadqueue{locality#0/total}/length,134,2.562788,[s],802
/threadqueue{locality#0/total}/length,135,2.580905,[s],803
/threadqueue{locality#0/total}/length,136,2.612951,[s],826
/threadqueue{locality#0/total}/length,137,2.626438,[s],737
/threadqueue{locality#0/total}/length,138,2.639359,[s],694
/threadqueue{locality#0/total}/length,139,2.652266,[s],724
/threadqueue{locality#0/total}/length,140,2.664968,[s],550
/threadqueue{locality#0/total}/length,141,2.677006,[s],355
/threadqueue{locality#0/total}/length,142,2.691321,[s],282
/threadqueue{locality#0/total}/length,143,2.705762,[s],286
/threadqueue{locality#0/total}/length,144,2.738461,[s],184
/threadqueue{locality#0/total}/length,145,2.758408,[s],352
/threadqueue{locality#0/total}/length,146,2.772511,[s],335
/threadqueue{locality#0/total}/length,147,2.790031,[s],167
/threadqueue{locality#0/total}/length,148,2.802457,[s],60
/threadqueue{locality#0/total}/length,149,2.814306,[s],0
/threadqueue{locality#0/total}/length,150,2.826177,[s],86
/threadqueue{locality#0/total}/length,151,2.846925,[s],304
/threadqueue{locality#0/total}/length,152,2.884310,[s],695
/threadqueue{locality#0/total}/length,153,2.902303,[s],692
/threadqueue{locality#0/total}/length,154,2.951843,[s],735
/threadqueue{locality#0/total}/length,155,2.983704,[s],609
/threadqueue{locality#0/total}/length,156,2.996253,[s],609
/threadqueue{locality#0/total}/length,157,3.010261,[s],595
/threadqueue{locality#0/total}/length,158,3.021584,[s],528
/threadqueue{locality#0/total}/length,159,3.034350,[s],464
/threadqueue{locality#0/total}/length,160,3.047527,[s],350
/threadqueue{locality#0/total}/length,161,3.060393,[s],272
/threadqueue{locality#0/total}/length,162,3.075251,[s],361
/threadqueue{locality#0/total}/length,163,3.100293,[s],596
/threadqueue{locality#0/total}/length,164,3.117238,[s],538
/threadqueue{locality#0/total}/length,165,3.130154,[s],387
/threadqueue{locality#0/total}/length,166,3.144063,[s],307
/threadqueue{locality#0/total}/length,167,3.156144,[s],267
/threadqueue{locality#0/total}/length,168,3.168601,[s],122
/threadqueue{locality#0/total}/length,169,3.180180,[s],22
/threadqueue{locality#0/total}/length,170,3.192659,[s],0
/threadqueue{locality#0/total}/length,171,3.210112,[s],147
/threadqueue{locality#0/total}/length,172,3.262345,[s],720
/threadqueue{locality#0/total}/length,173,3.306261,[s],809
/threadqueue{locality#0/total}/length,174,3.333300,[s],878
/threadqueue{locality#0/total}/length,175,3.362437,[s],1067
/threadqueue{locality#0/total}/length,176,3.375277,[s],1107
/threadqueue{locality#0/total}/length,177,3.392007,[s],1134
/threadqueue{locality#0/total}/length,178,3.405456,[s],1113
/threadqueue{locality#0/total}/length,179,3.423853,[s],951
/threadqueue{locality#0/total}/length,180,3.495884,[s],608
/threadqueue{locality#0/total}/length,181,3.509205,[s],495
/threadqueue{locality#0/total}/length,182,3.521515,[s],424
/threadqueue{locality#0/total}/length,183,3.535017,[s],317
/threadqueue{locality#0/total}/length,184,3.546451,[s],160
/threadqueue{locality#0/total}/length,185,3.560973,[s],8
/threadqueue{locality#0/total}/length,186,3.573426,[s],11
/threadqueue{locality#0/total}/length,187,3.593224,[s],219
/threadqueue{locality#0/total}/length,188,3.610533,[s],434
/threadqueue{locality#0/total}/length,189,3.642887,[s],823
/threadqueue{locality#0/total}/length,190,3.666908,[s],865
/threadqueue{locality#0/total}/length,191,3.699902,[s],935
/threadqueue{locality#0/total}/length,192,3.729209,[s],1140
/threadqueue{locality#0/total}/length,193,3.756062,[s],1330
/threadqueue{locality#0/total}/length,194,3.839158,[s],910
/threadqueue{locality#0/total}/length,195,3.872213,[s],728
/threadqueue{locality#0/total}/length,196,3.884270,[s],652
/threadqueue{locality#0/total}/length,197,3.898152,[s],532
/threadqueue{locality#0/total}/length,198,3.911764,[s],382
/threadqueue{locality#0/total}/length,199,3.926193,[s],209
/threadqueue{locality#0/total}/length,200,3.938400,[s],57
/threadqueue{locality#0/total}/length,201,3.949628,[s],0
100x100 0.0253616 MLUPS
/threadqueue{locality#0/total}/length,202,3.952730,[s],0

{stack-trace}: 18 frames:
0x7f7d5dd9b6c0  : hpx::util::backtrace::backtrace(unsigned long) + 0x80 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9b7ae  : hpx::util::trace_on_new_stack() + 0x1e in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd97fdc  : hpx::detail::backtrace() + 0x18 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9bb6d  : boost::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x46 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9c123  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x38 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd985be  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x238 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd98386  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x0 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5fda390a  : boost::any::any() + 0x0 in /home/wash/install/hpx/gcc-4.6.2-debug/bin/jacobi
0x7f7d5e022378  : hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) + 0x136 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e0219dd  : hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) + 0x4f7 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037f9f  : boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>::operator()(hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*, unsigned long, hpx::threads::topology const&) const + 0x6b in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037e28  : void boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> >::operator()<boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>&, boost::_bi::list0&, int) + 0x88 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037c93  : boost::_bi::bind_t<void, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> > >::operator()() + 0x39 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037b58  : boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> > > >::run() + 0x1e in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5ac23ceb  : ??? + 0x7f7d5ac23ceb in /opt/boost/1.52.0-debug/stage/lib/libboost_thread.so.1.52.0
0x7f7d591ffe0e  : ??? + 0x7f7d591ffe0e in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f7d58f3493d  : clone + 0x6d in /lib/x86_64-linux-gnu/libc.so.6
{env}: 37 entries:
  CPATH=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include
  DISPLAY=localhost:10.0
  EDITOR=vim
  GLIBCPP_FORCE_NEW=1
  GLIBCXX_FORCE_NEW=1
  HISTCONTROL=ignoredups
  HOME=/home/wash
  INCLUDE=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include
  INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses:/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses
  IPPROOT=/opt/intel/composer_xe_2013.1.117/ipp
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/visit/2.5.2/2.5.2/linux-x86-64/lib
  LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64
  LOGNAME=wash
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MAIL=/var/mail/wash
  MANPATH=/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/usr/local/man:/usr/local/share/man:/usr/share/man:::/opt/intel/vtune_amplifier_xe_2013/man
  MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic
  MKLROOT=/opt/intel/composer_xe_2013.1.117/mkl
  NLSPATH=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N
  OLDPWD=/srv/scratch/wash
  PATH=/opt/intel/vtune_amplifier_xe_2013/bin64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/visit/2.5.2/bin
  PWD=/srv/scratch/wash/hpx/gcc-4.6.2-debug
  SHELL=/bin/bash
  SHLVL=1
  SHOME=/srv/scratch/wash
  SILO_ROOT=/opt/silo/4.8-release
  SSH_CLIENT=10.1.1.1 56765 22
  SSH_CONNECTION=10.1.1.1 56765 10.1.1.13 22
  SSH_TTY=/dev/pts/0
  TBBROOT=/opt/intel/composer_xe_2013.1.117/tbb
  TERM=xterm
  USER=wash
  VISIT=/opt/visit/2.5.2/2.5.2/linux-x86-64
  VISITPLUGINDIR=/opt/visit/2.5.2/2.5.2/linux-x86-64/plugins
  VTUNE_AMPLIFIER_XE_2013_DIR=/opt/intel/vtune_amplifier_xe_2013
  _=/home/wash/install/hpx/gcc-4.6.2-debug/bin/jacobi
{locality-id}: 0
{process-id}: 11229
{function}: void hpx::threads::threadmanager_impl<SchedulingPolicy, NotificationPolicy>::tfunc_impl(std::size_t) [with SchedulingPolicy = hpx::threads::policies::local_priority_queue_scheduler<>, NotificationPolicy = hpx::threads::policies::callback_notifier, std::size_t = long unsigned int]
{file}: /home/wash/hpx/src/runtime/threads/threadmanager.cpp
{line}: 1318
{os-thread}: worker-thread#0
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_SHMEM=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=ON
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_LIMIT=5
  HPX_ACTION_ARGUMENT_LIMIT=4
  HPX_COMPONENT_CREATE_ARGUMENT_LIMIT=5
  HPX_FUNCTION_ARGUMENT_LIMIT=7
  HPX_LOCK_LIMIT=5
  HPX_TUPLE_LIMIT=10
  HPX_WAIT_ARGUMENT_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX=/home/wash/install/hpx/gcc-4.6.2-debug
{version}: V1.0.0-trunk (AGAS: V3.0), Git: d124c2eab3b774bdbd21752fd0895c314e145c60
{boost}: V1.52.0
{build-type}: debug
{date}: Sep 27 2013 10:53:42
{platform}: linux
{compiler}: GNU C++ version 4.6.4
{stdlib}: GNU libstdc++ version 20130412
{what}: assertion '!scheduler_.get_thread_count( unknown, thread_priority_default, num_thread)' failed: HPX(assertion_failure)

Aborted
brycelelbach commented 10 years ago

Hartmut: I'll test tonight to see if your second commit fixes this.

hkaiser commented 10 years ago

This seems to have been fixed with the recent commits