search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.51k stars 428 forks source link

Race condition during startup #982

Closed brycelelbach closed 10 years ago

brycelelbach commented 10 years ago 
[18:45:43]:wash@marvin00.hermione:/tmp:0:$ mpirun -x LD_LIBRARY_PATH="$LD_LIBRARY_PATH" -machinefile machinefile -bynode -np 16 /home/wash/install/octopus/intel-13.0.1-release/bin/3d_torus_driver --hpx:config /home/wash/sandbox/5lor_benchmark.ini -t16
    ___          
   (   \  \      
 /  \   \  | \   Octopus: a scalable HPX framework for AMR
 \__(0  0)_/ /   
   _//||\\__/       Copyright (c) 2012 Bryce Adelstein-Lelbach
  / | |\ \___/                         Zach Byerly
  \ | \ \__                            Dominic Marcello
    \_ \_        

Launching AMR driver...

[octopus]
levels_of_refinement                     = 5
runge_kutta_order                        = 3
reflect_on_z                             = false
spatial_domain                           = 1.500000e+00
grid_node_length                         = 14
temporal_domain                          = 1.000000e+00
temporal_prediction_gap                  = 10
output_frequency                         = 2.000000e+00
checkpoint_file                          = checkpoint_L%06u.bin
load_checkpoint                          = false

Found 16 usable localities, deploying infrastructure...
Initializing science table...

[octopus.3d_torus]
max_dt_growth                 = 1.250000
temporal_prediction_limiter   = 0.5
rotational_direction          = counterclockwise
momentum_conservation         = angular
rotating_grid                 = 1
kappa                         = 0.07852
X_in                          = 0.7
kick_mode                     = 3

R_0     = 0.823529
R_inner = 0.7
rho_max = 0.000218165
omega   = 1.33808
period  = 4.69568

Creating system components...
Executing application...

REFINEMENT PASS 1 OF 5, 9 NODES
REFINEMENT PASS 2 OF 5, 33 NODES
REFINEMENT PASS 3 OF 5, 97 NODES

{stack-trace}: 10 frames:
0x2aaaac45a273  : hpx::detail::backtrace() + 0x143 in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaac4620cd  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x3d in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaaca27597  : hpx::components::server::runtime_support::free_component(int, hpx::naming::gid_type const&, hpx::naming::gid_type const&) + 0x4b7 in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaaca9aaf9  : hpx::components::stubs::runtime_support::free_component_sync(int, hpx::naming::gid_type const&, hpx::naming::gid_type const&) + 0x79 in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaaca9aa7a  : hpx::components::stubs::runtime_support::free_component_sync(int, hpx::naming::gid_type const&, unsigned long) + 0x1a in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaacae6fe2  : hpx::naming::detail::decrement_refcnt(hpx::naming::detail::id_type_impl*) + 0x182 in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
0x2aaaaaf6bb00  : hpx::actions::typed_continuation<hpx::util::unused_type>::~typed_continuation() + 0x50 in /home/wash/install/octopus/intel-13.0.1-release/lib/hpx/lib3d_torus.so.0
0x2aaaab6a9c3d  : hpx::util::detail::vtable<false>::type_base<hpx::util::detail::bound_functor5<hpx::actions::action<octopus::octree_server, hpx::util::unused_type, hpx::util::tuple<octopus::face, octopus::octree_client, void, void, void, void, void, void, void, void, void, void, void, void, void, void, void, void>, hpx::actions::action2<void (octopus::octree_server::*)(octopus::face, octopus::octree_client const&), &octopus::octree_server::set_sibling, hpx::actions::detail::this_type> >::continuation_thread_object_function_void_2, boost::shared_ptr<hpx::actions::continuation>, void (octopus::octree_server::*)(octopus::face, octopus::octree_client const&), octopus::octree_server*, octopus::face, octopus::octree_client const> >::static_delete(void**) + 0x6d in /home/wash/install/octopus/intel-13.0.1-release/lib/hpx/liboctopus.so.0
0x2aaaac5d44fd  : void hpx::util::coroutines::detail::lx::trampoline<hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) + 0x64d in /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0
{env}: 98 entries:
  CPATH=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include
  DISPLAY=localhost:10.0
  EDITOR=vim
  GLIBCPP_FORCE_NEW=1
  GLIBCXX_FORCE_NEW=1
  HISTCONTROL=ignoredups
  HOME=/home/wash
  INCLUDE=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include
  INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses:/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses
  IPPROOT=/opt/intel/composer_xe_2013.1.117/ipp
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/visit/2.5.2/2.5.2/linux-x86-64/lib:/opt/visit/2.5.2/2.5.2/linux-x86-64/lib
  LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64
  LOGNAME=wash
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MAIL=/var/mail/wash
  MANPATH=/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/usr/local/man:/usr/local/share/man:/usr/share/man:::/opt/intel/vtune_amplifier_xe_2013/man
  MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic
  MKLROOT=/opt/intel/composer_xe_2013.1.117/mkl
  NLSPATH=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N
  OLDPWD=/home/wash
  OMPI_COMM_WORLD_LOCAL_RANK=0
  OMPI_COMM_WORLD_LOCAL_SIZE=1
  OMPI_COMM_WORLD_NODE_RANK=0
  OMPI_COMM_WORLD_RANK=7
  OMPI_COMM_WORLD_SIZE=16
  OMPI_MCA_ess=env
  OMPI_MCA_mpi_yield_when_idle=0
  OMPI_MCA_opal_cr_is_tool=0
  OMPI_MCA_orte_app_num=0
  OMPI_MCA_orte_cpu_model=Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz
  OMPI_MCA_orte_cpu_type=GenuineIntel
  OMPI_MCA_orte_ess_jobid=1434386433
  OMPI_MCA_orte_ess_node_rank=0
  OMPI_MCA_orte_ess_num_procs=16
  OMPI_MCA_orte_ess_vpid=7
  OMPI_MCA_orte_hnp_uri=1434386432.0;tcp://10.1.1.33:60269;tcp://192.168.1.7:60269
  OMPI_MCA_orte_local_daemon_uri=1434386432.7;tcp://10.1.1.40:58664;tcp://192.168.1.14:58664
  OMPI_MCA_orte_nodelist=marvin01,marvin02,marvin03,marvin04,marvin05,marvin06,marvin07,marvin08,marvin09,marvin10,marvin11,marvin12,marvin13,marvin14,marvin15
  OMPI_MCA_orte_num_nodes=16
  OMPI_MCA_orte_num_restarts=0
  OMPI_MCA_orte_precondition_transports=f8adae4d08b87079-d590e4a884625b8e
  OMPI_MCA_plm=rsh
  OMPI_MCA_shmem_RUNTIME_QUERY_hint=mmap
  OMPI_UNIVERSE_SIZE=16
  OPAL_OUTPUT_STDERR_FD=22
  PATH=/opt/intel/vtune_amplifier_xe_2013/bin64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/visit/2.5.2/bin:/opt/visit/2.5.2/bin
  PWD=/tmp
  SHELL=/bin/bash
  SHLVL=2
  SHOME=/srv/scratch/wash
  SILO_ROOT=/opt/silo/4.8-release
  SLURMD_NODENAME=marvin07
  SLURM_CHECKPOINT_IMAGE_DIR=/tmp
  SLURM_CPUS_ON_NODE=16
  SLURM_DISTRIBUTION=cyclic
  SLURM_GTIDS=6
  SLURM_JOBID=64045
  SLURM_JOB_CPUS_PER_NODE=16(x16)
  SLURM_JOB_ID=64045
  SLURM_JOB_NAME=/bin/bash
  SLURM_LAUNCH_NODE_IPADDR=10.1.1.33
  SLURM_LOCALID=0
  SLURM_NNODES=15
  SLURM_NODEID=6
  SLURM_NODELIST=marvin[00-15]
  SLURM_NPROCS=15
  SLURM_NTASKS=15
  SLURM_PRIO_PROCESS=0
  SLURM_PROCID=6
  SLURM_PTY_PORT=35253
  SLURM_PTY_WIN_COL=189
  SLURM_PTY_WIN_ROW=55
  SLURM_SRUN_COMM_HOST=10.1.1.33
  SLURM_SRUN_COMM_PORT=47565
  SLURM_STEPID=1
  SLURM_STEP_ID=1
  SLURM_STEP_LAUNCHER_PORT=47565
  SLURM_STEP_NODELIST=marvin[01-15]
  SLURM_STEP_NUM_NODES=15
  SLURM_STEP_NUM_TASKS=15
  SLURM_STEP_TASKS_PER_NODE=1(x15)
  SLURM_SUBMIT_DIR=/tmp
  SLURM_TASKS_PER_NODE=1(x15)
  SLURM_TASK_PID=11918
  SLURM_TOPOLOGY_ADDR=marvin07
  SLURM_TOPOLOGY_ADDR_PATTERN=node
  SSH_CLIENT=174.64.13.174 58531 8100
  SSH_CONNECTION=174.64.13.174 58531 130.39.22.145 8100
  SSH_TTY=/dev/pts/0
  TBBROOT=/opt/intel/composer_xe_2013.1.117/tbb
  TERM=xterm
  TMPDIR=/tmp
  USER=wash
  VISIT=/opt/visit/2.5.2/2.5.2/linux-x86-64
  VISITPLUGINDIR=/opt/visit/2.5.2/2.5.2/linux-x86-64/plugins
  VTUNE_AMPLIFIER_XE_2013_DIR=/opt/intel/vtune_amplifier_xe_2013
  _=/usr/bin/mpirun
{locality-id}: 7
{hostname}: 10.1.1.40:7917:7
{process-id}: 11925
{function}: runtime_support::free_component
{file}: /home/wash/hpx/src/runtime/components/server/runtime_support_server.cpp
{line}: 324
{os-thread}: 6, worker-thread#6
{thread-id}: 00002aaab3b407e0
{thread-description}: octopus_octree_server_set_sibling_action
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_SHMEM=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_LIMIT=5
  HPX_ACTION_ARGUMENT_LIMIT=4
  HPX_COMPONENT_CREATE_ARGUMENT_LIMIT=5
  HPX_FUNCTION_ARGUMENT_LIMIT=7
  HPX_LOCK_LIMIT=5
  HPX_TUPLE_LIMIT=10
  HPX_WAIT_ARGUMENT_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX=/home/wash/install/hpx/intel-13.0.1-release
{version}: V0.9.7-trunk (AGAS: V3.0), Git: 6012816e41476d72ca2742743efde5aca385dddf
{boost}: V1.53.0
{build-type}: release
{date}: Oct 22 2013 10:40:50
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1300
{stdlib}: GNU libstdc++ version 20130412
{what}: attempt to destroy component {0000000800ff0001, 0000000000301647} of invalid/unknown type: component[15(4)] (broadcast_symbol_namespace_service_action/component_base_lco_with_value[4])
list of registered components: 
  component_base_lco[3] (component_base_lco[3])
  component_barrier[5] (component_barrier[5])
  component_agas_locality_namespace[7] (component_agas_locality_namespace[7])
  component_agas_primary_namespace[8] (component_agas_primary_namespace[8])
  component_agas_component_namespace[9] (component_agas_component_namespace[9])
  component_agas_symbol_namespace[10] (component_agas_symbol_namespace[10])
  component[11] (base_performance_counter)
  component[12] (adding_counter)
  component[13] (average_count_counter)
  component[14] (binpacking_factory)
  component[15] (broadcast_symbol_namespace_service_action)
  component[16] (call_here_action)
  component[17] (console_error_sink_action)
  component[18] (console_logging_action)
  component[19] (console_print_action)
  component[20] (create_cerr_action)
  component[21] (create_cout_action)
  component[22] (dataflow)
  component[23] (dataflow_trigger)
  component[24] (distributing_factory)
  component[25] (dividing_counter)
  component[26] (elapsed_time_counter)
  component[27] (fstream_perform_start_write_action)
  component[28] (fstream_perform_stop_write_action)
  component[29] (hpx_get_locality_name_action)
  component[30] (list_component_type_action)
  component[31] (list_symbolic_name_action)
  component[32] (max_count_counter)
  component[33] (median_count_counter)
  component[34] (min_count_counter)
  component[35] (multiplying_counter)
  component[36] (notify_worker_action)
  component[37] (octopus_engine_server)
  component[38] (octopus_octree_server)
  component[39] (output_stream_factory)
  component[40] (papi_counter)
  component[41] (raw_counter)
  component[42] (register_worker_action)
  component[43] (remote_object)
  component[44] (remote_object_new_impl_action)
  component[45] (rolling_mean_count_counter)
  component[46] (silo_perform_start_write_action)
  component[47] (silo_perform_stop_write_action)
  component[48] (subtracting_counter)
  component[49] (variance_count_counter)
  component_barrier[5(3)] (component_barrier[5]/component_base_lco[3])
  component[12(4)] (adding_counter/component_base_lco_with_value[4])
  component[12(11)] (adding_counter/base_performance_counter)
  component[13(4)] (average_count_counter/component_base_lco_with_value[4])
  component[13(11)] (average_count_counter/base_performance_counter)
  component[14(4)] (binpacking_factory/component_base_lco_with_value[4])
  component[15(4)] (broadcast_symbol_namespace_service_action/component_base_lco_with_value[4])
  component[16(4)] (call_here_action/component_base_lco_with_value[4])
  component[17(4)] (console_error_sink_action/component_base_lco_with_value[4])
  component[18(4)] (console_logging_action/component_base_lco_with_value[4])
  component[22(3)] (dataflow/component_base_lco[3])
  component[23(3)] (dataflow_trigger/component_base_lco[3])
  component[25(11)] (dividing_counter/base_performance_counter)
  component[26(11)] (elapsed_time_counter/base_performance_counter)
  component[32(11)] (max_count_counter/base_performance_counter)
  component[33(11)] (median_count_counter/base_performance_counter)
  component[34(11)] (min_count_counter/base_performance_counter)
  component[35(11)] (multiplying_counter/base_performance_counter)
  component[40(11)] (papi_counter/base_performance_counter)
  component[41(11)] (raw_counter/base_performance_counter)
  component[45(11)] (rolling_mean_count_counter/base_performance_counter)
  component[48(11)] (subtracting_counter/base_performance_counter)
  component[49(11)] (variance_count_counter/base_performance_counter)
: HPX(bad_component_type)

[marvin07:11925] *** Process received signal ***
[marvin07:11925] Signal: Aborted (6)
[marvin07:11925] Signal code:  (-6)
[marvin07:11925] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf210) [0x2aaab0359210]
[marvin07:11925] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x2aaab059b1e5]
[marvin07:11925] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x2aaab059e398]
[marvin07:11925] [ 3] /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0(_ZN3hpx6detail30report_exception_and_terminateERKN5boost13exception_ptrE+0x40) [0x2aaaac454490]
[marvin07:11925] [ 4] /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexEEENS2_17callback_notifierEE12report_errorEmRKNS4_13exception_ptrE+0x2d) [0x2aaaac602c4d]
[marvin07:11925] [ 5] /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0(_ZN3hpx7threads18threadmanager_implINS0_8policies30local_priority_queue_schedulerIN5boost5mutexEEENS2_17callback_notifierEE12report_errorEmRKNS4_13exception_ptrE+0x5a) [0x2aaaaccb9dfa]
[marvin07:11925] [ 6] /home/wash/install/hpx/intel-13.0.1-release/lib/hpx/libhpx.so.0(_ZN3hpx7threads18threadmanager_implINS0_8policies30local_priority_queue_schedulerIN5boost5mutexEEENS2_17callback_notifierEE5tfuncEmRKNS0_8topologyE+0x852) [0x2aaaacc51dd2]
[marvin07:11925] [ 7] /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0(+0xe0fa) [0x2aaaaacb70fa]
[marvin07:11925] [ 8] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e0e) [0x2aaab0351e0e]
[marvin07:11925] [ 9] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x2aaab064e93d]
[marvin07:11925] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 11925 on node marvin07 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Happened once on top of the fixing_588 branch. Couldn't reproduce.

hkaiser commented 10 years ago

What should we do with this? Even more as this does not look like a startup problem.

hkaiser commented 10 years ago

Any new insights? If not, please close this ticket.

brycelelbach commented 10 years ago

I ran into this more frequently during SC. I believe this is related to after_588. Closing.