Open khuck opened 4 years ago
On the buildbot server, the distributed tcp runs work fine but the MPI runs don't. See http://omega.nic.uoregon.edu:8020/#/builders/5/builds/166/steps/16/logs/stdio for details.
This is openmpi 3.1.1 and gcc 7.3 on linux x86_64.
Here's the verbose output from one of the failed tests:
[khuck@delphi phylanx-Debug]$ ctest -V -R tests.unit.distributed.distributed.mpi.remote_run UpdateCTestConfiguration from :/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/DartConfiguration.tcl Parse Config file:/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/DartConfiguration.tcl UpdateCTestConfiguration from :/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/DartConfiguration.tcl Parse Config file:/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/DartConfiguration.tcl Test project /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph... Checking test dependency graph end test 138 Start 138: tests.unit.distributed.distributed.mpi.remote_run 138: Test command: /packages/python/3.6.4-ssl/bin/python3 "/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/phylanxrun.py" "/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test" "-e" "0" "-l" "2" "-t" "1" "-p" "mpi" "-r" "mpi" "-v" "--" 138: Test timeout computed to be: 200 138: 138: {config}: 138: Core library: 138: HPX_WITH_ACTION_BASE_COMPATIBILITY=ON 138: HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF 138: HPX_WITH_APEX=TRUE 138: HPX_WITH_APEX_NO_UPDATE=OFF 138: HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF 138: HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON 138: HPX_WITH_COROUTINE_COUNTERS=OFF 138: HPX_WITH_DEPRECATION_WARNINGS=ON 138: HPX_WITH_DISTRIBUTED_RUNTIME=ON 138: HPX_WITH_EMBEDDED_THREAD_POOLS_COMPATIBILITY=ON 138: HPX_WITH_GOOGLE_PERFTOOLS=OFF 138: HPX_WITH_HPXMP=OFF 138: HPX_WITH_INIT_START_OVERLOADS_COMPATIBILITY=ON 138: HPX_WITH_IO_COUNTERS=ON 138: HPX_WITH_IO_POOL=ON 138: HPX_WITH_ITTNOTIFY=OFF 138: HPX_WITH_LOGGING=ON 138: HPX_WITH_MORE_THAN_64_THREADS= 138: HPX_WITH_NATIVE_TLS=ON 138: HPX_WITH_NETWORKING=ON 138: HPX_WITH_PAPI=OFF 138: HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF 138: HPX_WITH_PARCELPORT_LIBFABRIC=OFF 138: HPX_WITH_PARCELPORT_MPI=ON 138: HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON 138: HPX_WITH_PARCELPORT_TCP=ON 138: HPX_WITH_PARCELPORT_VERBS=OFF 138: HPX_WITH_PARCEL_PROFILING=ON 138: HPX_WITH_REGISTER_THREAD_COMPATIBILITY=ON 138: HPX_WITH_REGISTER_THREAD_OVERLOADS_COMPATIBILITY=ON 138: HPX_WITH_SANITIZERS=OFF 138: HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF 138: HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF 138: HPX_WITH_STACKOVERFLOW_DETECTION=ON 138: HPX_WITH_STACKTRACES=ON 138: HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF 138: HPX_WITH_SWAP_CONTEXT_EMULATION=OFF 138: HPX_WITH_TESTS_DEBUG_LOG=OFF 138: HPX_WITH_THREAD_AWARE_TIMER_COMPATIBILITY=ON 138: HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF 138: HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF 138: HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON 138: HPX_WITH_THREAD_DEBUG_INFO=OFF 138: HPX_WITH_THREAD_DESCRIPTION_FULL=OFF 138: HPX_WITH_THREAD_EXECUTORS_COMPATIBILITY=ON 138: HPX_WITH_THREAD_GUARD_PAGE=ON 138: HPX_WITH_THREAD_IDLE_RATES=ON 138: HPX_WITH_THREAD_LOCAL_STORAGE=OFF 138: HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=OFF 138: HPX_WITH_THREAD_POOL_OS_EXECUTOR_COMPATIBILITY=ON 138: HPX_WITH_THREAD_QUEUE_WAITTIME=OFF 138: HPX_WITH_THREAD_STACK_MMAP=ON 138: HPX_WITH_THREAD_STEALING_COUNTS=OFF 138: HPX_WITH_THREAD_TARGET_ADDRESS=OFF 138: HPX_WITH_TIMER_POOL=ON 138: HPX_WITH_TUPLE_RVALUE_SWAP=ON 138: HPX_WITH_VALGRIND=OFF 138: HPX_WITH_VERIFY_LOCKS=OFF 138: HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF 138: HPX_WITH_VERIFY_LOCKS_GLOBALLY=OFF 138: 138: HPX_PARCEL_MAX_CONNECTIONS=512 138: HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4 138: HPX_AGAS_LOCAL_CACHE_SIZE=4096 138: HPX_HAVE_MALLOC=tcmalloc 138: HPX_PREFIX (configured)= 138: HPX_PREFIX=/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug 138: 138: Module affinity: 138: HPX_AFFINITY_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_AFFINITY_WITH_DEPRECATION_WARNINGS=ON 138: HPX_AFFINITY_WITH_TESTS=OFF 138: 138: Module algorithms: 138: HPX_ALGORITHMS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ALGORITHMS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ALGORITHMS_WITH_TESTS=OFF 138: 138: Module allocator_support: 138: HPX_ALLOCATOR_SUPPORT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ALLOCATOR_SUPPORT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ALLOCATOR_SUPPORT_WITH_TESTS=OFF 138: 138: Module asio: 138: HPX_ASIO_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ASIO_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ASIO_WITH_TESTS=OFF 138: 138: Module assertion: 138: HPX_ASSERTION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ASSERTION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ASSERTION_WITH_TESTS=OFF 138: 138: Module async: 138: HPX_ASYNC_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ASYNC_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ASYNC_WITH_TESTS=OFF 138: 138: Module async_base: 138: HPX_ASYNC_BASE_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_ASYNC_BASE_WITH_TESTS=OFF 138: 138: Module async_combinators: 138: HPX_ASYNC_COMBINATORS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ASYNC_COMBINATORS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ASYNC_COMBINATORS_WITH_TESTS=OFF 138: 138: Module basic_execution: 138: HPX_BASIC_EXECUTION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_BASIC_EXECUTION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_BASIC_EXECUTION_WITH_TESTS=OFF 138: 138: Module batch_environments: 138: HPX_BATCH_ENVIRONMENTS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_BATCH_ENVIRONMENTS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_BATCH_ENVIRONMENTS_WITH_TESTS=OFF 138: 138: Module cache: 138: HPX_CACHE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_CACHE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_CACHE_WITH_TESTS=OFF 138: 138: Module checkpoint: 138: HPX_CHECKPOINT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_CHECKPOINT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_CHECKPOINT_WITH_TESTS=OFF 138: 138: Module collectives: 138: HPX_COLLECTIVES_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_COLLECTIVES_WITH_DEPRECATION_WARNINGS=ON 138: HPX_COLLECTIVES_WITH_TESTS=OFF 138: 138: Module command_line_handling: 138: HPX_COMMAND_LINE_HANDLING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_COMMAND_LINE_HANDLING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_COMMAND_LINE_HANDLING_WITH_TESTS=OFF 138: 138: Module compute: 138: HPX_COMPUTE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_COMPUTE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_COMPUTE_WITH_TESTS=OFF 138: 138: Module compute_cuda: 138: HPX_COMPUTE_CUDA_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_COMPUTE_CUDA_WITH_TESTS=OFF 138: 138: Module concepts: 138: HPX_CONCEPTS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_CONCEPTS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_CONCEPTS_WITH_TESTS=OFF 138: 138: Module concurrency: 138: HPX_CONCURRENCY_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_CONCURRENCY_WITH_DEPRECATION_WARNINGS=ON 138: HPX_CONCURRENCY_WITH_TESTS=OFF 138: 138: Module config: 138: HPX_CONFIG_WITH_COMPATIBILITY_HEADERS=OFF 138: HPX_CONFIG_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_CONFIG_WITH_TESTS=OFF 138: 138: Module config_registry: 138: HPX_CONFIG_REGISTRY_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_CONFIG_REGISTRY_WITH_TESTS=OFF 138: 138: Module coroutines: 138: HPX_COROUTINES_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_COROUTINES_WITH_DEPRECATION_WARNINGS=ON 138: HPX_COROUTINES_WITH_TESTS=OFF 138: 138: Module datastructures: 138: HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON 138: HPX_DATASTRUCTURES_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_DATASTRUCTURES_WITH_DEPRECATION_WARNINGS=ON 138: HPX_DATASTRUCTURES_WITH_TESTS=OFF 138: 138: Module debugging: 138: HPX_DEBUGGING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_DEBUGGING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_DEBUGGING_WITH_TESTS=OFF 138: 138: Module distributed_executors: 138: HPX_DISTRIBUTED_EXECUTORS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_DISTRIBUTED_EXECUTORS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_DISTRIBUTED_EXECUTORS_WITH_TESTS=OFF 138: 138: Module errors: 138: HPX_ERRORS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ERRORS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ERRORS_WITH_TESTS=OFF 138: 138: Module execution: 138: HPX_EXECUTION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_EXECUTION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_EXECUTION_WITH_TESTS=OFF 138: 138: Module executors: 138: HPX_EXECUTORS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_EXECUTORS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_EXECUTORS_WITH_TESTS=OFF 138: 138: Module filesystem: 138: HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=ON 138: HPX_FILESYSTEM_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_FILESYSTEM_WITH_TESTS=OFF 138: 138: Module format: 138: HPX_FORMAT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_FORMAT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_FORMAT_WITH_TESTS=OFF 138: 138: Module functional: 138: HPX_FUNCTIONAL_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_FUNCTIONAL_WITH_DEPRECATION_WARNINGS=ON 138: HPX_FUNCTIONAL_WITH_TESTS=OFF 138: 138: Module hardware: 138: HPX_HARDWARE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_HARDWARE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_HARDWARE_WITH_TESTS=OFF 138: 138: Module hashing: 138: HPX_HASHING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_HASHING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_HASHING_WITH_TESTS=OFF 138: 138: Module include: 138: HPX_INCLUDE_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_INCLUDE_WITH_TESTS=OFF 138: 138: Module io_service: 138: HPX_IO_SERVICE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_IO_SERVICE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_IO_SERVICE_WITH_TESTS=OFF 138: 138: Module iterator_support: 138: HPX_ITERATOR_SUPPORT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_ITERATOR_SUPPORT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_ITERATOR_SUPPORT_WITH_TESTS=OFF 138: 138: Module lcos_distributed: 138: HPX_LCOS_DISTRIBUTED_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_LCOS_DISTRIBUTED_WITH_DEPRECATION_WARNINGS=ON 138: HPX_LCOS_DISTRIBUTED_WITH_TESTS=OFF 138: 138: Module local_async: 138: HPX_LOCAL_ASYNC_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_LOCAL_ASYNC_WITH_TESTS=OFF 138: 138: Module local_lcos: 138: HPX_LOCAL_LCOS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_LOCAL_LCOS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_LOCAL_LCOS_WITH_TESTS=OFF 138: 138: Module logging: 138: HPX_LOGGING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_LOGGING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_LOGGING_WITH_TESTS=OFF 138: 138: Module memory: 138: HPX_MEMORY_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_MEMORY_WITH_TESTS=OFF 138: 138: Module mpi: 138: HPX_MPI_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_MPI_WITH_FUTURES=OFF 138: HPX_MPI_WITH_TESTS=OFF 138: 138: Module mpi_base: 138: HPX_MPI_BASE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_MPI_BASE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_MPI_BASE_WITH_TESTS=OFF 138: 138: Module naming_base: 138: HPX_NAMING_BASE_WITH_COMPATIBILITY_HEADERS=OFF 138: HPX_NAMING_BASE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_NAMING_BASE_WITH_TESTS=OFF 138: 138: Module pack_traversal: 138: HPX_PACK_TRAVERSAL_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_PACK_TRAVERSAL_WITH_DEPRECATION_WARNINGS=ON 138: HPX_PACK_TRAVERSAL_WITH_TESTS=OFF 138: 138: Module performance_counters: 138: HPX_PERFORMANCE_COUNTERS_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_PERFORMANCE_COUNTERS_WITH_TESTS=OFF 138: 138: Module plugin: 138: HPX_PLUGIN_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_PLUGIN_WITH_DEPRECATION_WARNINGS=ON 138: HPX_PLUGIN_WITH_TESTS=OFF 138: 138: Module prefix: 138: HPX_PREFIX_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_PREFIX_WITH_DEPRECATION_WARNINGS=ON 138: HPX_PREFIX_WITH_TESTS=OFF 138: 138: Module preprocessor: 138: HPX_PREPROCESSOR_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_PREPROCESSOR_WITH_DEPRECATION_WARNINGS=ON 138: HPX_PREPROCESSOR_WITH_TESTS=OFF 138: 138: Module program_options: 138: HPX_PROGRAM_OPTIONS_WITH_BOOST_PROGRAM_OPTIONS_COMPATIBILITY=ON 138: HPX_PROGRAM_OPTIONS_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_PROGRAM_OPTIONS_WITH_TESTS=OFF 138: 138: Module resiliency: 138: HPX_RESILIENCY_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_RESILIENCY_WITH_TESTS=OFF 138: 138: Module resource_partitioner: 138: HPX_RESOURCE_PARTITIONER_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_RESOURCE_PARTITIONER_WITH_DEPRECATION_WARNINGS=ON 138: HPX_RESOURCE_PARTITIONER_WITH_TESTS=OFF 138: 138: Module runtime_configuration: 138: HPX_RUNTIME_CONFIGURATION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_RUNTIME_CONFIGURATION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_RUNTIME_CONFIGURATION_WITH_TESTS=OFF 138: 138: Module schedulers: 138: HPX_SCHEDULERS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_SCHEDULERS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_SCHEDULERS_WITH_TESTS=OFF 138: 138: Module segmented_algorithms: 138: HPX_SEGMENTED_ALGORITHMS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_SEGMENTED_ALGORITHMS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_SEGMENTED_ALGORITHMS_WITH_TESTS=OFF 138: 138: Module serialization: 138: HPX_SERIALIZATION_WITH_BOOST_TYPES=ON 138: HPX_SERIALIZATION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_SERIALIZATION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_SERIALIZATION_WITH_TESTS=OFF 138: 138: Module static_reinit: 138: HPX_STATIC_REINIT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_STATIC_REINIT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_STATIC_REINIT_WITH_TESTS=OFF 138: 138: Module statistics: 138: HPX_STATISTICS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_STATISTICS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_STATISTICS_WITH_TESTS=OFF 138: 138: Module string_util: 138: HPX_STRING_UTIL_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_STRING_UTIL_WITH_TESTS=OFF 138: 138: Module synchronization: 138: HPX_SYNCHRONIZATION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_SYNCHRONIZATION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_SYNCHRONIZATION_WITH_TESTS=OFF 138: 138: Module testing: 138: HPX_TESTING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_TESTING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_TESTING_WITH_TESTS=OFF 138: 138: Module thread_executors: 138: HPX_THREAD_EXECUTORS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREAD_EXECUTORS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREAD_EXECUTORS_WITH_TESTS=OFF 138: 138: Module thread_pools: 138: HPX_THREAD_POOLS_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREAD_POOLS_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREAD_POOLS_WITH_TESTS=OFF 138: 138: Module thread_support: 138: HPX_THREAD_SUPPORT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREAD_SUPPORT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREAD_SUPPORT_WITH_TESTS=OFF 138: 138: Module threading: 138: HPX_THREADING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREADING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREADING_WITH_TESTS=OFF 138: 138: Module threading_base: 138: HPX_THREADING_BASE_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREADING_BASE_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREADING_BASE_WITH_TESTS=OFF 138: 138: Module threadmanager: 138: HPX_THREADMANAGER_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_THREADMANAGER_WITH_DEPRECATION_WARNINGS=ON 138: HPX_THREADMANAGER_WITH_TESTS=OFF 138: 138: Module timed_execution: 138: HPX_TIMED_EXECUTION_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_TIMED_EXECUTION_WITH_DEPRECATION_WARNINGS=ON 138: HPX_TIMED_EXECUTION_WITH_TESTS=OFF 138: 138: Module timing: 138: HPX_TIMING_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_TIMING_WITH_DEPRECATION_WARNINGS=ON 138: HPX_TIMING_WITH_TESTS=OFF 138: 138: Module topology: 138: HPX_TOPOLOGY_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_TOPOLOGY_WITH_DEPRECATION_WARNINGS=ON 138: HPX_TOPOLOGY_WITH_TESTS=OFF 138: 138: Module type_support: 138: HPX_TYPE_SUPPORT_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_TYPE_SUPPORT_WITH_DEPRECATION_WARNINGS=ON 138: HPX_TYPE_SUPPORT_WITH_TESTS=OFF 138: 138: Module util: 138: HPX_UTIL_WITH_COMPATIBILITY_HEADERS=ON 138: HPX_UTIL_WITH_DEPRECATION_WARNINGS=ON 138: HPX_UTIL_WITH_TESTS=OFF 138: 138: Module version: 138: HPX_VERSION_WITH_COMPATIBILITY_HEADERS=OFF 138: HPX_VERSION_WITH_DEPRECATION_WARNINGS=OFF 138: HPX_VERSION_WITH_TESTS=OFF 138: 138: {version}: V1.5.0-trunk (AGAS: V3.0), Git: 62c825ff76 138: {boost}: V1.65.0 138: {build-type}: debug 138: {date}: May 15 2020 11:55:04 138: {platform}: linux 138: {compiler}: GNU C++ version 7.3.0 138: {stdlib}: GNU libstdc++ version 20180125 138: {env}: 155 entries: 138: ARCHDIR=x86_64 138: BASH_ENV=/usr/share/lmod/lmod/init/bash 138: BASH_FUNC_ml()=() { eval $($LMOD_DIR/ml_cmd "$@") 138: } 138: BASH_FUNC_module()=() { eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh) 138: } 138: BLAS_DIR=/usr/local/packages/lapack/3.7.1-gcc 138: BLAS_ROOT=/usr/local/packages/lapack/3.7.1-gcc 138: BOOST_DIR=/usr/local/packages/boost/1.65.0-gcc7 138: BOOST_ROOT=/usr/local/packages/boost/1.65.0-gcc7 138: CMAKE=/usr/local/packages/cmake/3.15.1 138: CPLUS_INCLUDE_PATH=/packages/gcc/7.3/include:/packages/openmpi/3.1.1-gcc7.3/include 138: CTEST_INTERACTIVE_DEBUG_MODE=1 138: CVSROOT=khuck@ix.cs.uoregon.edu:/research/paraducks2/cvs-src/master 138: CVS_RSH=ssh 138: C_INCLUDE_PATH=/packages/gcc/7.3/include:/packages/openmpi/3.1.1-gcc7.3/include 138: EDITOR=vim 138: GAMESS=/home/users/khuck/src/gamess/gamess 138: GCC=/packages/gcc/7.3 138: GMP=/packages/gmp/6.1.2 138: HDF5_LDFLAGS=-L/usr/local/packages/szip/2.1/gcc-4.9/lib -ldl -lz -lsz 138: HDF5_ROOT=/usr/local/packages/hdf5/1.10.1-gcc-6.4 138: HFI_NO_BACKTRACE=1 138: HISTCONTROL=ignoredups 138: HISTSIZE=1000 138: HOME=/home/users/khuck 138: HOSTNAME=delphi 138: INFOPATH=/packages/gmp/6.1.2/share/info:/packages/mpfr/3.1.6/share/info:/packages/mpc/1.0.3/share/info:/packages/gcc/7.3/info 138: IPATH_NO_BACKTRACE=1 138: ISL=/packages/isl/0.18 138: JAVA_HOME=/home/users/khuck/build/jdk1.6.0_29 138: LANG=en_US.UTF-8 138: LAPACK_DIR=/usr/local/packages/lapack/3.7.1-gcc 138: LAPACK_ROOT=/usr/local/packages/lapack/3.7.1-gcc 138: LD_LIBRARY_PATH=/packages/openmpi/3.1.1-gcc7.3/lib:/packages/openmpi/3.1.1-gcc7.3/lib:/packages/python/3.6.4-ssl/lib:/packages/isl/0.18/lib:/packages/gmp/6.1.2/lib:/packages/mpfr/3.1.6/lib:/packages/mpc/1.0.3/lib:/packages/gcc/7.3/lib64:/packages/gcc/7.3/lib:/packages/openmpi/3.1.1-gcc7.3/lib::/home/db2inst1/sqllib/lib:/home/users/khuck/tools/lib:/opt/R/lib/R/library/SJava/libs:/opt/R/lib/R/lib:/home/users/khuck/install/libunwind-1.1-gnu/lib 138: LESSOPEN=||/usr/bin/lesspipe.sh %s 138: LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod 138: LMOD_DIR=/usr/share/lmod/lmod/libexec 138: LMOD_PKG=/usr/share/lmod/lmod 138: LMOD_ROOT=/usr/share/lmod 138: LMOD_SETTARG_FULL_SUPPORT=no 138: LMOD_VERSION=8.2.7 138: LMOD_sys=Linux 138: LOADEDMODULES=gcc/7.3:mpi/openmpi-3.1.1_gcc-7.3:python/3.6.4:cmake/3.15.1 138: LOGNAME=khuck 138: LS_COLORS=:no=00:fi=00:di=36:ln=01;33:pi=33:so=01;35:do=01;35:bd=01;37:cd=01;37:ex=01;32:or=01;37;41:mi=01;37;41:*.tar=01;31:*.tgz=01;31:*.gz=01;31:*.zip=01;31:*.sit=01;31:*.Z=01;31:*.bz2=01;31:*.z=01;31:*.rpm=01;31 138: MAIL=/var/spool/mail/khuck 138: MANPATH=/usr/local/packages/cmake/3.15.1/man:/packages/python/3.6.4-ssl/man:/packages/gcc/7.3/share/man:/packages/openmpi/3.1.1-gcc7.3/share/man:/usr/share/lmod/lmod/share/man:: 138: MODULEPATH=/usr/local/packages/Modules/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles 138: MODULEPATH_ROOT=/usr/share/modulefiles 138: MODULESHOME=/usr/share/lmod/lmod 138: MPC=/packages/mpc/1.0.3 138: MPFR=/packages/mpfr/3.1.6 138: MPI=/packages/openmpi/3.1.1-gcc7.3 138: MPICC=mpicc 138: MPICXX=mpicxx 138: MPI_LOCALNRANKS=2 138: MPI_LOCALRANKID=1 138: OLDPWD=/home/users/khuck/src/phylanx/tools/buildbot 138: OMPI_APP_CTX_NUM_PROCS=2 138: OMPI_ARGV=--hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.bootstrap=mpi --hpx:threads=1 --hpx:localities=2 138: OMPI_COMMAND=remote_run_test 138: OMPI_COMM_WORLD_LOCAL_RANK=1 138: OMPI_COMM_WORLD_LOCAL_SIZE=2 138: OMPI_COMM_WORLD_NODE_RANK=1 138: OMPI_COMM_WORLD_RANK=1 138: OMPI_COMM_WORLD_SIZE=2 138: OMPI_FILE_LOCATION=/tmp/ompi.delphi.14978/pid.21184/0/0 138: OMPI_FIRST_RANKS=0 138: OMPI_MCA_ess=pmi 138: OMPI_MCA_ess_base_jobid=1663762433 138: OMPI_MCA_ess_base_vpid=1 138: OMPI_MCA_initial_wdir=/storage/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/tests/unit/distributed 138: OMPI_MCA_mpi_yield_when_idle=0 138: OMPI_MCA_orte_app_num=0 138: OMPI_MCA_orte_bound_at_launch=1 138: OMPI_MCA_orte_ess_node_rank=1 138: OMPI_MCA_orte_ess_num_procs=2 138: OMPI_MCA_orte_hnp_uri=1663762432.0;tcp://172.17.202.67,128.223.202.67,172.18.0.1,172.24.0.1:60329;ud://797258.4.1 138: OMPI_MCA_orte_jobfam_session_dir=/tmp/ompi.delphi.14978/pid.21184 138: OMPI_MCA_orte_launch=1 138: OMPI_MCA_orte_local_daemon_uri=1663762432.0;tcp://172.17.202.67,128.223.202.67,172.18.0.1,172.24.0.1:60329;ud://797258.4.1 138: OMPI_MCA_orte_num_nodes=1 138: OMPI_MCA_orte_precondition_transports=a9826274055cfd18-532945a57607ca5f 138: OMPI_MCA_orte_tmpdir_base=/tmp 138: OMPI_MCA_orte_top_session_dir=/tmp/ompi.delphi.14978 138: OMPI_MCA_pmix=^s1,s2,cray,isolated 138: OMPI_MCA_shmem_RUNTIME_QUERY_hint=mmap 138: OMPI_NUM_APP_CTX=1 138: OMPI_UNIVERSE_SIZE=36 138: OPENSS_PLUGIN_PATH=/usr/local/lib64/openspeedshop/ 138: PATH=/packages/openmpi/3.1.1-gcc7.3/bin:/packages/openmpi/3.1.1-gcc7.3/bin:/usr/local/packages/cmake/3.15.1/bin:/packages/python/3.6.4-ssl/bin:/packages/gcc/7.3/bin:/packages/openmpi/3.1.1-gcc7.3/bin:/home/users/khuck/build/jdk1.6.0_29/bin:/home/users/khuck/local/bin:/home/users/khuck/install/visit2_10_2.linux-x86_64/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/users/khuck/src/tau2/x86_64/bin:/home/db2inst1/sqllib/bin:/home/users/khuck/install/htop/2.0.2/bin:/home/users/khuck/tools/bin:. 138: PKG_CONFIG_PATH=:/usr/local/packages/lapack/3.7.1-gcc/lib64/pkgconfig 138: PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_NON_DESC 138: PMIX_DSTORE_ESH_BASE_PATH=/tmp/ompi.delphi.14978/pid.21184/pmix_dstor_21184 138: PMIX_GDS_MODULE=ds12,hash 138: PMIX_ID=1663762433.1 138: PMIX_NAMESPACE=1663762433 138: PMIX_PTL_MODULE=tcp,usock 138: PMIX_RANK=1 138: PMIX_SECURITY_MODE=native,none 138: PMIX_SERVER_TMPDIR=/tmp/ompi.delphi.14978/pid.21184/0/0 138: PMIX_SERVER_URI21=1663762432.0;tcp4://127.0.0.1:50936 138: PMIX_SERVER_URI2=1663762432.0;tcp4://127.0.0.1:50936 138: PSM2_DEVICES=self,shm 138: PSM2_MULTI_EP=1 138: PWD=/storage/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/tests/unit/distributed 138: PYTHON=/packages/python/3.6.4-ssl/bin/python3.6 138: PYTHONHOME=/packages/python/3.6.4-ssl 138: PYTHONTOP=/packages/python/3.6.4-ssl 138: SELINUX_LEVEL_REQUESTED= 138: SELINUX_ROLE_REQUESTED= 138: SELINUX_USE_CURRENT_RANGE= 138: SHELL=/bin/bash 138: SHLVL=1 138: SSH_AUTH_SOCK=/tmp/ssh-p0gBVZN8iu/agent.53580 138: SSH_CLIENT=128.223.202.148 50618 22 138: SSH_CONNECTION=128.223.202.148 50618 128.223.202.67 22 138: SSH_TTY=/dev/pts/3 138: SVNROOT=http://precog.uoregon.edu/svn 138: TAU_MPI=-mpiinc=/packages/openmpi/3.1.1-gcc7.3/include -mpilib=/packages/openmpi/3.1.1-gcc7.3/lib 138: TERM=xterm 138: USER=khuck 138: XDG_DATA_DIRS=/home/users/khuck/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share 138: XDG_RUNTIME_DIR=/run/user/14978 138: XDG_SESSION_ID=17919 138: _=/usr/local/packages/cmake/3.15.1/bin/ctest 138: _LMFILES_=/usr/local/packages/Modules/modulefiles/gcc/7.3:/usr/local/packages/Modules/modulefiles/mpi/openmpi-3.1.1_gcc-7.3:/usr/local/packages/Modules/modulefiles/python/3.6.4:/usr/local/packages/Modules/modulefiles/cmake/3.15.1 138: _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtjbWFrZT17WyJmbiJdPSIvdXNyL2xvY2FsL3BhY2thZ2VzL01vZHVsZXMvbW9kdWxlZmlsZXMvY21ha2UvMy4xNS4xIixbImZ1bGxOYW1lIl09ImNtYWtlLzMuMTUuMSIsWyJsb2FkT3JkZXIiXT00LHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09ImNtYWtlIix9LGdjYz17WyJmbiJdPSIvdXNyL2xvY2FsL3BhY2thZ2VzL01vZHVsZXMvbW9kdWxlZmlsZXMvZ2NjLzcuMyIsWyJmdWxsTmFtZSJdPSJnY2MvNy4zIixbImxvYWRPcmRlciJdPTEs 138: _ModuleTable002_=cHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MSxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iZ2NjLzcuMyIsfSxtcGk9e1siZm4iXT0iL3Vzci9sb2NhbC9wYWNrYWdlcy9Nb2R1bGVzL21vZHVsZWZpbGVzL21waS9vcGVubXBpLTMuMS4xX2djYy03LjMiLFsiZnVsbE5hbWUiXT0ibXBpL29wZW5tcGktMy4xLjFfZ2NjLTcuMyIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09Im1waS9vcGVubXBpLTMuMS4xX2djYy03LjMiLH0scHl0aG9uPXtbImZuIl09Ii91c3IvbG9jYWwvcGFja2FnZXMvTW9kdWxlcy9tb2R1bGVmaWxlcy9weXRob24vMy42LjQiLFsiZnVsbE5hbWUiXT0icHl0 138: _ModuleTable003_=aG9uLzMuNi40IixbImxvYWRPcmRlciJdPTMscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0icHl0aG9uLzMuNi40Iix9LH0sbXBhdGhBPXsiL3Vzci9sb2NhbC9wYWNrYWdlcy9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL2V0Yy9tb2R1bGVmaWxlcyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvdXNyL2xvY2FsL3BhY2thZ2VzL01vZHVsZXMvbW9kdWxlZmlsZXM6L3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzOi9ldGMvbW9kdWxlZmlsZXMiLH0= 138: _ModuleTable_Sz_=3 138: __LMOD_REF_COUNT_CPLUS_INCLUDE_PATH=/packages/gcc/7.3/include:1;/packages/openmpi/3.1.1-gcc7.3/include:1 138: __LMOD_REF_COUNT_C_INCLUDE_PATH=/packages/gcc/7.3/include:1;/packages/openmpi/3.1.1-gcc7.3/include:1 138: __LMOD_REF_COUNT_INFOPATH=/packages/gmp/6.1.2/share/info:1;/packages/mpfr/3.1.6/share/info:1;/packages/mpc/1.0.3/share/info:1;/packages/gcc/7.3/info:1 138: __LMOD_REF_COUNT_LD_LIBRARY_PATH=/packages/python/3.6.4-ssl/lib:1;/packages/isl/0.18/lib:1;/packages/gmp/6.1.2/lib:1;/packages/mpfr/3.1.6/lib:1;/packages/mpc/1.0.3/lib:1;/packages/gcc/7.3/lib64:1;/packages/gcc/7.3/lib:1;/packages/openmpi/3.1.1-gcc7.3/lib:1;/home/db2inst1/sqllib/lib:1;/home/users/khuck/tools/lib:1;/opt/R/lib/R/library/SJava/libs:1;/opt/R/lib/R/lib:1;/home/users/khuck/install/libunwind-1.1-gnu/lib:1 138: __LMOD_REF_COUNT_LOADEDMODULES=gcc/7.3:1;mpi/openmpi-3.1.1_gcc-7.3:1;python/3.6.4:1;cmake/3.15.1:1 138: __LMOD_REF_COUNT_MANPATH=/usr/local/packages/cmake/3.15.1/man:1;/packages/python/3.6.4-ssl/man:1;/packages/gcc/7.3/share/man:1;/packages/openmpi/3.1.1-gcc7.3/share/man:1;/usr/share/lmod/lmod/share/man:1 138: __LMOD_REF_COUNT_MODULEPATH=/usr/local/packages/Modules/modulefiles:1;/usr/share/Modules/modulefiles:1;/etc/modulefiles:1 138: __LMOD_REF_COUNT_PATH=/usr/local/packages/cmake/3.15.1/bin:1;/packages/python/3.6.4-ssl/bin:1;/packages/gcc/7.3/bin:1;/packages/openmpi/3.1.1-gcc7.3/bin:1;/home/users/khuck/build/jdk1.6.0_29/bin:1;/home/users/khuck/local/bin:1;/home/users/khuck/install/visit2_10_2.linux-x86_64/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/users/khuck/src/tau2/x86_64/bin:1;/home/db2inst1/sqllib/bin:1;/home/users/khuck/install/htop/2.0.2/bin:1;/home/users/khuck/tools/bin:1;.:1 138: __LMOD_REF_COUNT__LMFILES_=/usr/local/packages/Modules/modulefiles/gcc/7.3:1;/usr/local/packages/Modules/modulefiles/mpi/openmpi-3.1.1_gcc-7.3:1;/usr/local/packages/Modules/modulefiles/python/3.6.4:1;/usr/local/packages/Modules/modulefiles/cmake/3.15.1:1 138: activeharmony_path=/usr/local/packages/activeharmony/4.6.0-gcc 138: basedir=/home/users/khuck/src/phylanx/tools 138: boost_path=/usr/local/packages/boost/1.65.0-gcc7 138: buildtype=Release 138: host=delphi 138: malloc=tcmalloc 138: malloc_path=/usr/local/packages/gperftools/git-gcc7 138: myarch=delphi-x86_64-Linux-gcc 138: mycc=gcc 138: mycflags=-fPIC 138: mycxx=g++ 138: mycxxflags=-fPIC 138: myfc=gfortran 138: myldflags=-fPIC -Wl,--allow-multiple-definition 138: otf2_path=/usr/local/packages/otf2-2.1 138: papi_path=/usr/local/packages/papi/papi-knl/5.5.0/ 138: {stack-trace}: 18 frames: 138: 0x7f8cdcdf94e9 : hpx::detail::custom_exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xdd] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdce2d080 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x673080) [0x7f8cdce2d080] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd59cea2 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xde2ea2) [0x7f8cdd59cea2] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd59c3fb : std::__exception_ptr::exception_ptr hpx::detail::construct_custom_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xa2] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd59b370 : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x77] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdce6b284 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x6b1284) [0x7f8cdce6b284] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd4c9879 : hpx::assertion::detail::handle_assert(hpx::assertion::source_location const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xee] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd004b01 : hpx::agas::addressing_service::launch_bootstrap(std::shared_ptr<hpx::parcelset::parcelport> const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) [0x295] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd004653 : hpx::agas::addressing_service::bootstrap(hpx::parcelset::parcelhandler&, hpx::util::runtime_configuration const&) [0x15b] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdd41404b : hpx::runtime_distributed::runtime_distributed(hpx::util::runtime_configuration&) [0x4ab] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x7f8cdce19f06 : hpx::detail::run_or_start(hpx::util::function<int (boost::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&, bool) [0x29a] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1 138: 0x4a1e23 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test() [0x4a1e23] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test 138: 0x4a1e9c : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test() [0x4a1e9c] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test 138: 0x49d343 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test() [0x49d343] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test 138: 0x4bc87a : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test() [0x4bc87a] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test 138: 0x7f8cd982a505 : __libc_start_main [0xf5] in /lib64/libc.so.6 138: 0x49ca59 : /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test() [0x49ca59] in /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test 138: {locality-id}: 0 138: {hostname}: [ (mpi:1) (tcp:127.0.0.1:7910) ] 138: {process-id}: 21191 138: {os-thread}: <unknown> 138: {thread-description}: <anonymous task> 138: {state}: state_initialized 138: {auxinfo}: 138: {file}: /home/users/khuck/src/phylanx/tools/buildbot/src/hpx/src/runtime/agas/addressing_service.cpp 138: {line}: 254 138: {function}: void hpx::agas::addressing_service::launch_bootstrap(const std::shared_ptr<hpx::parcelset::parcelport>&, const endpoints_type&, const hpx::util::runtime_configuration&) 138: {what}: Assertion 'pp ? pp->here() == pp->agas_locality(cfg) : true' failed: HPX(assertion_failure) 138: 138: ------------------------------------------------------- 138: Primary job terminated normally, but 1 process returned 138: a non-zero exit code. Per user-direction, the job has been aborted. 138: ------------------------------------------------------- 138: -------------------------------------------------------------------------- 138: mpiexec detected that one or more processes exited with non-zero status, thus causing 138: the job to be terminated. The first process to do so was: 138: 138: Process name: [[25387,1],1] 138: Exit code: 255 138: -------------------------------------------------------------------------- 138: Base command is "/home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.bootstrap=mpi --hpx:threads=1 --hpx:localities=2" 138: Executing command: /packages/openmpi/3.1.1-gcc7.3/bin/mpiexec -n 2 /home/users/khuck/src/phylanx/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/remote_run_test --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.bootstrap=mpi --hpx:threads=1 --hpx:localities=2 1/1 Test #138: tests.unit.distributed.distributed.mpi.remote_run ...***Failed 4.06 sec 0% tests passed, 1 tests failed out of 1 Total Test time (real) = 4.19 sec The following tests FAILED: 138 - tests.unit.distributed.distributed.mpi.remote_run (Failed) Errors while running CTest
On the buildbot server, the distributed tcp runs work fine but the MPI runs don't. See http://omega.nic.uoregon.edu:8020/#/builders/5/builds/166/steps/16/logs/stdio for details.
This is openmpi 3.1.1 and gcc 7.3 on linux x86_64.
Here's the verbose output from one of the failed tests: