search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.5k stars 429 forks source link

building with parcelport_mpi fails on cori #2873

Closed kleinhenz closed 7 years ago

kleinhenz commented 7 years ago

Building on cori with parcelport_mpi currently fails to compile mpi_environment.cpp with numerous errors mostly complaining about mpi_environment not being declared or not having a type. I am using gcc-6.3.0 with cray compiler wrapper, cmake 3.8.2, boost 1.62.

compiler output: https://pastebin.com/raw/0FrD1j13

build script:

m -rf build && mkdir build && cd build &&                    \
cmake                                                         \
 -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/CrayStatic.cmake  \
 -DCMAKE_INSTALL_PREFIX=~/local/stow/hpx                      \
 -DHPX_WITH_STATIC_LINKING=ON                                 \
 -DHPX_WITH_STATIC_EXE_LINKING=ON                             \
 -DCMAKE_BUILD_TYPE=Release                                   \
 -DHWLOC_ROOT=$HOME/local                                     \
 -DHPX_WITH_MALLOC="system"                                   \
 -DHPX_WITH_PARCELPORT_TCP=ON                                 \
 -DHPX_WITH_PARCELPORT_MPI=ON                                 \
 -DHPX_WITH_PARCELPORT_LIBFABRIC=OFF                          \
 -DHPX_WITH_CXX14=ON                                          \
 -DHPX_WITH_EXAMPLES=OFF                                      \
 -DHPX_WITH_TESTS=ON                                          \
 -DHPX_WITH_TESTS_BENCHMARKS=OFF                              \
 -DHPX_WITH_TESTS_EXTERNAL_BUILD=OFF                          \
 -DHPX_WITH_TESTS_HEADERS=OFF                                 \
 -DHPX_WITH_TESTS_REGRESSIONS=OFF                             \
 -DHPX_WITH_TESTS_UNIT=ON                                     \
..

my currently loaded modules

Currently Loaded Modulefiles:
  1) modules/3.2.10.6                               7) cray-mpich/7.6.0                              13) dmapp/7.1.1-6.0.4.0_46.2__gb8abda2.ari        19) rca/2.2.11-6.0.4.0_13.2__g84de67a.ari         25) python/3.6-anaconda-4.4
  2) nsg/1.2.0                                      8) cray-shmem/7.6.0                              14) gni-headers/5.0.11-6.0.4.0_7.2__g7136988.ari  20) atp/2.1.1                                     26) tmux/2.2
  3) gcc/6.3.0                                      9) cray-libsci/17.06.1                           15) xpmem/2.2.2-6.0.4.0_3.1__g43b0535.ari         21) PrgEnv-gnu/6.0.4                              27) git/2.9.1
  4) craype-haswell                                10) udreg/2.3.2-6.0.4.0_12.2__g2f9c3ee.ari        16) job/2.2.2-6.0.4.0_8.2__g3c644b5.ari           22) eigen3/3.3.3                                  28) cmake/3.8.2
  5) craype-network-aries                          11) ugni/6.0.14-6.0.4.0_14.1__ge7db4a2.ari        17) dvs/2.7_2.2.31-6.0.4.1_6.1__gb3b87e6          23) cray-hdf5/1.8.16                              29) altd/2.0
  6) craype/2.5.12                                 12) pmi/5.0.12                                    18) alps/6.4.1-6.0.4.0_7.2__g86d0f3d.ari          24) boost/1.62                                    30) darshan/3.1.4
hkaiser commented 7 years ago

Could you please post the full output from the cmake step as well?

kleinhenz commented 7 years ago

output of configure step

ModuleCmd_Switch.c(179):ERROR:152: Module 'PrgEnv-intel/6.0.4' is currently not loaded
-- The CXX compiler identification is GNU 6.3.0
-- The C compiler identification is GNU 6.3.0
-- Cray Programming Environment 2.5.12 CXX
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Cray Programming Environment 2.5.12 C
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- No vectorization library configured
-- Native TLS is enabled.
-- Nice threadlevel is disabled.
-- Stack traces are enabled.
-- Compiling with the native toolset
-- C++ mode enforced: C++14
-- Performing Test HPX_WITH_CXX11_ALIAS_TEMPLATES - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_AUTO - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_CONSTEXPR - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_DECLTYPE - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_SFINAE_EXPRESSION - Success
-- Performing Test HPX_WITH_CXX11_DEFAULTED_FUNCTIONS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_DELETED_FUNCTIONS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_EXPLICIT_CONVERSION_OPERATORS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_EXPLICIT_VARIADIC_TEMPLATES - Success
-- Performing Test HPX_WITH_CXX11_EXTENDED_FRIEND_DECLARATIONS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_FUNCTION_TEMPLATE_DEFAULT_ARGS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_INLINE_NAMESPACES - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_LAMBDAS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_NOEXCEPT - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_NULLPTR - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_NSDMI - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_RANGE_BASED_FOR - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_RVALUE_REFERENCES - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_SCOPED_ENUMS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_STATIC_ASSERT - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_VARIADIC_MACROS - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_VARIADIC_TEMPLATES - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX11_ARRAY - Success
-- Performing Test HPX_WITH_CXX11_ATOMIC - Success
-- Performing Test HPX_WITH_CXX11_CHRONO - Success
-- Performing Test HPX_WITH_CXX11_CSTDINT - Success
-- Performing Test HPX_WITH_CXX11_EXCEPTION_PTR - Success
-- Performing Test HPX_WITH_CXX11_STD_INITIALIZER_LIST - Success
-- Performing Test HPX_WITH_CXX11_IS_BIND_EXPRESSION - Success
-- Performing Test HPX_WITH_CXX11_IS_PLACEHOLDER - Success
-- Performing Test HPX_WITH_CXX11_IS_TRIVIALLY_COPYABLE - Success
-- Performing Test HPX_WITH_CXX11_LOCK_GUARD - Success
-- Performing Test HPX_WITH_CXX11_RANGE_ACCESS - Success
-- Performing Test HPX_WITH_CXX11_REFERENCE_WRAPPER - Success
-- Performing Test HPX_WITH_CXX11_SHARED_PTR - Success
-- Performing Test HPX_WITH_CXX11_SHUFFLE - Success
-- Performing Test HPX_WITH_CXX11_THREAD - Success
-- Performing Test HPX_WITH_CXX11_TO_STRING - Success
-- Performing Test HPX_WITH_CXX11_UNIQUE_LOCK - Success
-- Performing Test HPX_WITH_CXX11_UNIQUE_PTR - Success
-- Performing Test HPX_WITH_CXX11_UNORDERED_MAP - Success
-- Performing Test HPX_WITH_CXX11_UNORDERED_SET - Success
-- Performing Test HPX_WITH_CXX11_NORETURN_ATTRIBUTE - Success
-- Performing Test HPX_WITH_CXX14_CONSTEXPR - Success
-- Performing Test HPX_WITH_CXX14_LAMBDAS - Success
-- Performing Test HPX_WITH_CXX14_INTEGER_SEQUENCE - Success
-- Performing Test HPX_WITH_CXX14_IS_FINAL - Success
-- Performing Test HPX_WITH_CXX14_IS_NULL_POINTER - Success
-- Performing Test HPX_WITH_CXX14_RESULT_OF_SFINAE - Success
-- Performing Test HPX_WITH_CXX14_VARIABLE_TEMPLATES - Success (cmake feature test)
-- Performing Test HPX_WITH_CXX14_DEPRECATED_ATTRIBUTE - Success
-- Performing Test HPX_WITH_CXX14_RETURN_TYPE_DEDUCTION - Success (cmake feature test)
-- Performing Test HPX_WITH_LIBFUN_EXPERIMENTAL_OPTIONAL - Success
-- Performing Test HPX_WITH_MM_PREFECTH - Success
-- Performing Test HPX_WITH_UNISTD_H - Success
-- Performing Test HPX_WITH_CXX_FLAG_WALL
-- Performing Test HPX_WITH_CXX_FLAG_WALL - Success
-- Performing Test HPX_WITH_C_FLAG_WALL
-- Performing Test HPX_WITH_C_FLAG_WALL - Success
-- Performing Test HPX_WITH_Fortran_FLAG_WALL
-- Performing Test HPX_WITH_Fortran_FLAG_WALL - Success
-- Performing Test HPX_WITH_CXX_FLAG_WEXTRA
-- Performing Test HPX_WITH_CXX_FLAG_WEXTRA - Success
-- Performing Test HPX_WITH_C_FLAG_WEXTRA
-- Performing Test HPX_WITH_C_FLAG_WEXTRA - Success
-- Performing Test HPX_WITH_Fortran_FLAG_WEXTRA
-- Performing Test HPX_WITH_Fortran_FLAG_WEXTRA - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS - Success
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS - Success
-- Performing Test HPX_WITH_Fortran_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS
-- Performing Test HPX_WITH_Fortran_FLAG_WNO_UNUSED_LOCAL_TYPEDEFS - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_STRICT_ALIASING
-- Performing Test HPX_WITH_CXX_FLAG_WNO_STRICT_ALIASING - Success
-- Performing Test HPX_WITH_C_FLAG_WNO_STRICT_ALIASING
-- Performing Test HPX_WITH_C_FLAG_WNO_STRICT_ALIASING - Success
-- Performing Test HPX_WITH_Fortran_FLAG_WNO_STRICT_ALIASING
-- Performing Test HPX_WITH_Fortran_FLAG_WNO_STRICT_ALIASING - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_SIGN_PROMO
-- Performing Test HPX_WITH_CXX_FLAG_WNO_SIGN_PROMO - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_ATTRIBUTES
-- Performing Test HPX_WITH_CXX_FLAG_WNO_ATTRIBUTES - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_CAST_ALIGN
-- Performing Test HPX_WITH_CXX_FLAG_WNO_CAST_ALIGN - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_PARAMETER
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_PARAMETER - Success
-- Performing Test HPX_WITH_CXX_FLAG_WFORMAT_2
-- Performing Test HPX_WITH_CXX_FLAG_WFORMAT_2 - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_FORMAT_NONLITERAL
-- Performing Test HPX_WITH_CXX_FLAG_WNO_FORMAT_NONLITERAL - Success
-- Performing Test HPX_WITH_CXX_FLAG_WINIT_SELF
-- Performing Test HPX_WITH_CXX_FLAG_WINIT_SELF - Success
-- Performing Test HPX_WITH_CXX_FLAG_WDOUBLE_PROMOTION
-- Performing Test HPX_WITH_CXX_FLAG_WDOUBLE_PROMOTION - Success
-- Performing Test HPX_WITH_CXX_FLAG_WCAST_QUAL
-- Performing Test HPX_WITH_CXX_FLAG_WCAST_QUAL - Success
-- Performing Test HPX_WITH_CXX_FLAG_WCAST_ALIGN
-- Performing Test HPX_WITH_CXX_FLAG_WCAST_ALIGN - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_TRAMPOLINES
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_TRAMPOLINES - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_PARENTHESES
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_PARENTHESES - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_REORDER
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_REORDER - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_RETURN_TYPE
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_RETURN_TYPE - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_SEQUENCE_POINT
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_SEQUENCE_POINT - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_UNINITIALIZED
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_UNINITIALIZED - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_FORMAT
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_FORMAT - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_MISSING_BRACES
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_MISSING_BRACES - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_SIGN_COMPARE
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_SIGN_COMPARE - Success
-- Performing Test HPX_WITH_CXX_FLAG_FDIAGNOSTICS_SHOW_OPTION
-- Performing Test HPX_WITH_CXX_FLAG_FDIAGNOSTICS_SHOW_OPTION - Success
-- Performing Test HPX_WITH_C_FLAG_FDIAGNOSTICS_SHOW_OPTION
-- Performing Test HPX_WITH_C_FLAG_FDIAGNOSTICS_SHOW_OPTION - Success
-- Performing Test HPX_WITH_Fortran_FLAG_FDIAGNOSTICS_SHOW_OPTION
-- Performing Test HPX_WITH_Fortran_FLAG_FDIAGNOSTICS_SHOW_OPTION - Success
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_VLA
-- Performing Test HPX_WITH_CXX_FLAG_WERROR_VLA - Success
-- Performing Test HPX_WITH_C_FLAG_WERROR_RETURN_TYPE
-- Performing Test HPX_WITH_C_FLAG_WERROR_RETURN_TYPE - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_BUT_SET_PARAMETER
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_BUT_SET_PARAMETER - Success
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_BUT_SET_PARAMETER
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_BUT_SET_PARAMETER - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_BUT_SET_VARIABLE
-- Performing Test HPX_WITH_CXX_FLAG_WNO_UNUSED_BUT_SET_VARIABLE - Success
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_BUT_SET_VARIABLE
-- Performing Test HPX_WITH_C_FLAG_WNO_UNUSED_BUT_SET_VARIABLE - Success
-- Performing Test HPX_WITH_C_FLAG_WERROR_UNINITIALIZED
-- Performing Test HPX_WITH_C_FLAG_WERROR_UNINITIALIZED - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_SYNC_NAND
-- Performing Test HPX_WITH_CXX_FLAG_WNO_SYNC_NAND - Success
-- Performing Test HPX_WITH_C_FLAG_WNO_SYNC_NAND
-- Performing Test HPX_WITH_C_FLAG_WNO_SYNC_NAND - Success
-- Performing Test HPX_WITH_CXX_FLAG_WNO_DELETE_NON_VIRTUAL_DTOR
-- Performing Test HPX_WITH_CXX_FLAG_WNO_DELETE_NON_VIRTUAL_DTOR - Success
-- Performing Test WITH_LINKER_FLAG_WL__Z_DEFS
-- Performing Test WITH_LINKER_FLAG_WL__Z_DEFS - Success
-- Performing Test HPX_WITH_CXX_FLAG_FVISIBILITY_HIDDEN
-- Performing Test HPX_WITH_CXX_FLAG_FVISIBILITY_HIDDEN - Success
-- Performing Test HPX_WITH_C_FLAG_FVISIBILITY_HIDDEN
-- Performing Test HPX_WITH_C_FLAG_FVISIBILITY_HIDDEN - Success
-- Performing Test HPX_WITH_Fortran_FLAG_FVISIBILITY_HIDDEN
-- Performing Test HPX_WITH_Fortran_FLAG_FVISIBILITY_HIDDEN - Success
-- Performing Test WITH_LINKER_FLAG_FVISIBILITY_HIDDEN
-- Performing Test WITH_LINKER_FLAG_FVISIBILITY_HIDDEN - Success
-- Performing Test HPX_WITH_RDTSC - Success
-- Performing Test HPX_WITH_RDTSCP - pre-set to ON
-- Boost version: 1.62.0
-- Found the following Boost libraries:
--   atomic
--   filesystem
--   program_options
--   regex
--   system
--   random
-- WARNING: HPX will perform poorly without tcmalloc or jemalloc. See docs for more info.
-- Using system allocator.
-- Found Hwloc: /global/homes/j/jkleinh/local/lib/libhwloc.a
-- Found Git: /usr/common/software/git/2.9.1/bin/git (found version "2.9.1")
-- GIT commit is 56ff70a14bfeae1fd757b7782fd0e84207bdc8da.
-- Performing Test HPX_WITH_CXX11_SFINAE_EXPRESSION_COMPLETE - Success
-- Found MPI_C: /opt/cray/pe/craype/2.5.12/bin/cc
-- Found MPI_CXX: /opt/cray/pe/craype/2.5.12/bin/CC
-- Found PythonInterp: /usr/common/software/python/3.6-anaconda-4.4/bin/python (found version "3.6.1")

-- Configuration summary:
--   HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
--   HPX_WITH_ALGORITHM_INPUT_ITERATOR_SUPPORT=OFF
--   HPX_WITH_APEX=OFF
--   HPX_WITH_ASYNC_FUNCTION_COMPATIBILITY=OFF
--   HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
--   HPX_WITH_BOOST_CHRONO_COMPATIBILITY=OFF
--   HPX_WITH_COLOCATED_BACKWARDS_COMPATIBILITY=OFF
--   HPX_WITH_COMPILER_WARNINGS=ON
--   HPX_WITH_COMPILE_ONLY_TESTS=ON
--   HPX_WITH_COMPONENT_GET_GID_COMPATIBILITY=OFF
--   HPX_WITH_COMPRESSION_BZIP2=OFF
--   HPX_WITH_COMPRESSION_SNAPPY=OFF
--   HPX_WITH_COMPRESSION_ZLIB=OFF
--   HPX_WITH_CUDA=OFF
--   HPX_WITH_CUDA_CLANG=OFF
--   HPX_WITH_CXX14_RETURN_TYPE_DEDUCTION=TRUE
--   HPX_WITH_DATAPAR_BOOST_SIMD=OFF
--   HPX_WITH_DATAPAR_VC=OFF
--   HPX_WITH_DEFAULT_TARGETS=ON
--   HPX_WITH_DEPRECATION_WARNINGS=ON
--   HPX_WITH_DISABLED_SIGNAL_EXCEPTION_HANDLERS=OFF
--   HPX_WITH_DOCUMENTATION=OFF
--   HPX_WITH_DOCUMENTATION_SINGLEPAGE=OFF
--   HPX_WITH_EXAMPLES=OFF
--   HPX_WITH_EXAMPLES_HDF5=OFF
--   HPX_WITH_EXAMPLES_OPENMP=OFF
--   HPX_WITH_EXAMPLES_QT4=OFF
--   HPX_WITH_EXAMPLES_QTHREADS=OFF
--   HPX_WITH_EXAMPLES_TBB=OFF
--   HPX_WITH_EXECUTION_POLICY_COMPATIBILITY=ON
--   HPX_WITH_EXECUTOR_COMPATIBILITY=ON
--   HPX_WITH_FORTRAN=OFF
--   HPX_WITH_FULL_RPATH=ON
--   HPX_WITH_GCC_VERSION_CHECK=ON
--   HPX_WITH_GENERIC_CONTEXT_COROUTINES=OFF
--   HPX_WITH_GENERIC_EXECUTION_POLICY=OFF
--   HPX_WITH_GOOGLE_PERFTOOLS=OFF
--   HPX_WITH_HCC=OFF
--   HPX_WITH_HIDDEN_VISIBILITY=ON
--   HPX_WITH_HWLOC=ON
--   HPX_WITH_INCLUSIVE_SCAN_COMPATIBILITY=ON
--   HPX_WITH_IO_COUNTERS=ON
--   HPX_WITH_IO_POOL=ON
--   HPX_WITH_ITTNOTIFY=OFF
--   HPX_WITH_LOCAL_DATAFLOW_COMPATIBILITY=ON
--   HPX_WITH_LOGGING=ON
--   HPX_WITH_MALLOC=system
--   HPX_WITH_MAX_CPU_COUNT=64
--   HPX_WITH_MORE_THAN_64_THREADS=(
--   HPX_WITH_NATIVE_TLS=ON
--   HPX_WITH_NETWORKING=ON
--   HPX_WITH_NICE_THREADLEVEL=OFF
--   HPX_WITH_PAPI=OFF
--   HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
--   HPX_WITH_PARCELPORT_LIBFABRIC=OFF
--   HPX_WITH_PARCELPORT_MPI=ON
--   HPX_WITH_PARCELPORT_MPI_ENV=MV2_COMM_WORLD_RANK PMI_RANK OMPI_COMM_WORLD_SIZE ALPS_APP_PE
--   HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON
--   HPX_WITH_PARCELPORT_TCP=ON
--   HPX_WITH_PARCELPORT_VERBS=OFF
--   HPX_WITH_PARCEL_COALESCING=ON
--   HPX_WITH_PARCEL_PROFILING=OFF
--   HPX_WITH_PSEUDO_DEPENDENCIES=ON
--   HPX_WITH_QUEUE_COMPATIBILITY=OFF
--   HPX_WITH_RUNTIME=ON
--   HPX_WITH_RUN_MAIN_EVERYWHERE=OFF
--   HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
--   HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
--   HPX_WITH_STACKOVERFLOW_DETECTION=OFF
--   HPX_WITH_STACKTRACES=ON
--   HPX_WITH_STATIC_EXE_LINKING=ON
--   HPX_WITH_STATIC_LINKING=ON
--   HPX_WITH_SWAP_CONTEXT_EMULATION=OFF
--   HPX_WITH_SYCL=OFF
--   HPX_WITH_TESTS=ON
--   HPX_WITH_TESTS_BENCHMARKS=OFF
--   HPX_WITH_TESTS_EXTERNAL_BUILD=OFF
--   HPX_WITH_TESTS_HEADERS=OFF
--   HPX_WITH_TESTS_REGRESSIONS=OFF
--   HPX_WITH_TESTS_UNIT=ON
--   HPX_WITH_THREAD_BACKTRACE_DEPTH=5
--   HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
--   HPX_WITH_THREAD_COMPATIBILITY=OFF
--   HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
--   HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
--   HPX_WITH_THREAD_DEBUG_INFO=OFF
--   HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
--   HPX_WITH_THREAD_GUARD_PAGE=ON
--   HPX_WITH_THREAD_IDLE_RATES=OFF
--   HPX_WITH_THREAD_LOCAL_STORAGE=OFF
--   HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
--   HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
--   HPX_WITH_THREAD_SCHEDULERS=all
--   HPX_WITH_THREAD_STACKOVERFLOW_DETECTION=OFF
--   HPX_WITH_THREAD_STACK_MMAP=ON
--   HPX_WITH_THREAD_STEALING_COUNTS=ON
--   HPX_WITH_THREAD_TARGET_ADDRESS=OFF
--   HPX_WITH_TIMER_POOL=ON
--   HPX_WITH_TOOLS=OFF
--   HPX_WITH_TRANSFORM_REDUCE_COMPATIBILITY=ON
--   HPX_WITH_TUPLE_RVALUE_SWAP=ON
--   HPX_WITH_UNWRAPPED_COMPATIBILITY=ON
--   HPX_WITH_VALGRIND=OFF
--   HPX_WITH_VERIFY_LOCKS=OFF
--   HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
--   HPX_WITH_VERIFY_LOCKS_GLOBALLY=OFF
--   HPX_WITH_VIM_YCM=OFF
--   HPX_WITH_ZERO_COPY_SERIALIZATION_THRESHOLD=4096
failed to create symbolic link '/global/homes/j/jkleinh/research/libraries/hpx/build/share/hpx': No such file or directory

HPX will be installed to /global/homes/j/jkleinh/local/stow/hpx

-- Configuring done
-- Generating done
-- Build files have been written to: /global/homes/j/jkleinh/research/libraries/hpx/build
hkaiser commented 7 years ago

The only explanation I have is that you have another HPX version installed somewhere and the compiler finds the wrong file hpx/config/defines.hpp.

Please have a look at /global/homes/j/jkleinh/research/libraries/hpx/build/hpx/config/defines.hpp which should have a #define HPX_HAVE_PARCELPORT_MPI to verify that the correct configuration setting were generated by cmake.

kleinhenz commented 7 years ago

I checked and I do have #define HPX_HAVE_PARCELPORT_MPI in the correct config file.

hkaiser commented 7 years ago

That confirms that in your environment the compiler finds another hpx/config/defines.hpp which does not have this configuration setting enabled. Please look carefully where that might be. Running make VERBOSE=1 gives you the executed command lines which might give a hint for where to look.

sithhell commented 7 years ago

It looks like your environment/build directory is a little screwed. I suggest to start from scratch, with an empty build directory. One very suspicious line is that one here:

ModuleCmd_Switch.c(179):ERROR:152: Module 'PrgEnv-intel/6.0.4' is currently not loaded

From the build log. This shouldn't happen. Also, you should remove darshan from your environment.

kleinhenz commented 7 years ago

As I understand it the ModuleCmd_switch.c error is emitted by the Cray compiler wrappers when not using PrgEnv-intel. I have successfully build other applications including hpx without mpi/jemalloc with PrgEnv-gnu and this warning hasn't been a problem. I am using PrgEnv-gnu because there were other compiler errors when compiling hpx with the intel compiler and I have successfully built hpx on a local machine with gcc. The build script wipes the build directory so each build is totally fresh so that shouldn't be the problem.

Maybe this is the wrong issue to open. The directions for building hpx on the nersc website no longer work so the current environment is the result of several days of debugging. I can start from a clean environment and post the errors I get from there, or do you want to wait until 1.1.0 feature freeze to stabilize code on all environments?

hkaiser commented 7 years ago

@kleinhenz I don't think waiting would help you in this case. I agree with Thomas that something is wrong in your environment.

kleinhenz commented 7 years ago

ok so I think I found the problem that I was having. The mpi compilation errors were coming from finding the wrong config file like you thought. My original problem was coming from the hwloc dependency. The default build of libhwloc.a is not fully static and includes dynamic dependencies on a bunch of stuff, e.g. libnuma, libxml2, etc. This means that these dependencies have to be provided when building hpx but findpackage(Hwloc) does not resolve these transitive dependencies so the build fails at the linking stage. I fixed the issue by following these directions to generate a fully static build of libhwloc.a. I found some discussion of this issue in the gromacs project. I'm not sure if it should be hpx's responsibility to fulfill the transitive dependencies of hwloc or if hpx should just require a fully static hwloc when building statically.

I can produce the problem on cori starting from a clean environment with this build script

module switch PrgEnv-intel/6.0.4 PrgEnv-gnu/6.0.4
module load cray-mpich/7.6.0
module load boost/1.62
module load hwloc/1.11.4
module load cmake/3.8.2
module unload darshan

rm -rf build && mkdir build && cd build &&                    \
cmake                                                         \
 -DCMAKE_BUILD_TYPE=Release                                   \
 -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/CrayStatic.cmake  \
 -DHPX_WITH_STATIC_LINKING=ON                                 \
 -DHPX_WITH_STATIC_EXE_LINKING=ON                             \
 -DCMAKE_INSTALL_PREFIX=$HOME/local/stow/hpx                  \
 -DHWLOC_ROOT=$HWLOC_DIR                                      \
 -DHPX_WITH_MALLOC="system"                                   \
 -DHPX_WITH_PARCELPORT_TCP=ON                                 \
 -DHPX_WITH_PARCELPORT_MPI=ON                                 \
 -DHPX_WITH_PARCELPORT_LIBFABRIC=OFF                          \
 -DHPX_WITH_CXX14=ON                                          \
 -DHPX_WITH_EXAMPLES=OFF                                      \
 -DHPX_WITH_TESTS=ON                                          \
 -DHPX_WITH_TESTS_BENCHMARKS=OFF                              \
 -DHPX_WITH_TESTS_EXTERNAL_BUILD=OFF                          \
 -DHPX_WITH_TESTS_HEADERS=OFF                                 \
 -DHPX_WITH_TESTS_REGRESSIONS=OFF                             \
 -DHPX_WITH_TESTS_UNIT=ON                                     \
.. && make -j 16                                              \

The build fails with the following error

[100%] Linking CXX executable ../bin/hpx_runtime
cd /global/homes/j/jkleinh/code/spike/hpx/build/runtime && /global/common/cori/software/cmake/3.8.2/bin/cmake -E cmake_link_script CMakeFiles/hpx_runtime_exe.dir/link.txt --verbose=1
/opt/cray/pe/craype/2.5.12/bin/CC -fPIC  -Wall -Wextra -Wno-unused-local-typedefs -Wno-strict-aliasing -Wno-sign-promo -Wno-attributes -Wno-cast-align -Wno-unused-parameter -Wformat=2 -Wno-format-nonliteral -Winit-self -Wdouble-promotion -Wcast-qual -Wcast-align -Werror=trampolines -Werror=parentheses -Werror=reorder -Werror=return-type -Werror=sequence-point -Werror=uninitialized -Werror=format -Werror=missing-braces -Werror=sign-compare -fdiagnostics-show-option -Werror=vla -Werror=return-type -Wno-unused-but-set-parameter -Wno-unused-but-set-variable -Werror=uninitialized -Wno-unused-local-typedefs -Wno-sync-nand -Wno-delete-non-virtual-dtor -fvisibility=hidden -O3 -DNDEBUG   -static -Wl,-z,defs -fvisibility=hidden CMakeFiles/hpx_runtime_exe.dir/hpx_runtime.cpp.o -o ../bin/hpx_runtime  -L/global/homes/j/jkleinh/code/spike/hpx/build/lib  -L/usr/common/software/boost/1.62/gnu/haswell/lib -Wl,-rpath,/global/homes/j/jkleinh/code/spike/hpx/build/lib:/usr/common/software/boost/1.62/gnu/haswell/lib: ../lib/libhpx.a -latomic -ldl -lrt /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_atomic.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_filesystem.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_program_options.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_regex.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_system.a /usr/lib64/libpthread.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_atomic.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_filesystem.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_program_options.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_regex.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_system.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_random.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_system.a /global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_atomic.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_filesystem.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_program_options.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_regex.a /usr/lib64/libpthread.a /usr/common/software/boost/1.62/gnu/haswell/lib/libboost_random.a /global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a
../lib/libhpx.a(runtime_support_server.cpp.o): In function `hpx::util::plugin::dll::LoadLibrary(hpx::error_code&, bool) [clone .constprop.3224]':
runtime_support_server.cpp:(.text+0x1a94): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
../lib/libhpx.a(asio_util.cpp.o): In function `boost::asio::ip::basic_resolver<boost::asio::ip::tcp, boost::asio::ip::resolver_service<boost::asio::ip::tcp> >::resolve(boost::asio::ip::basic_resolver_query<boost::asio::ip::tcp> const&)':
asio_util.cpp:(.text._ZN5boost4asio2ip14basic_resolverINS1_3tcpENS1_16resolver_serviceIS3_EEE7resolveERKNS1_20basic_resolver_queryIS3_EE[_ZN5boost4asio2ip14basic_resolverINS1_3tcpENS1_16resolver_serviceIS3_EEE7resolveERKNS1_20basic_resolver_queryIS3_EE]+0x185): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml2_disable_stderrwarnings':
topology-xml-libxml.c:(.text+0xa7): undefined reference to `__xmlGenericError'
topology-xml-libxml.c:(.text+0xc0): undefined reference to `xmlSetGenericErrorFunc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_look_init':
topology-xml-libxml.c:(.text+0x4c6): undefined reference to `xmlGetIntSubset'
topology-xml-libxml.c:(.text+0x560): undefined reference to `xmlDocGetRootElement'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_import_diff':
topology-xml-libxml.c:(.text+0x6ca): undefined reference to `xmlCheckVersion'
topology-xml-libxml.c:(.text+0x6f7): undefined reference to `xmlReadFile'
topology-xml-libxml.c:(.text+0x725): undefined reference to `xmlReadMemory'
topology-xml-libxml.c:(.text+0x75c): undefined reference to `xmlGetIntSubset'
topology-xml-libxml.c:(.text+0x7f2): undefined reference to `xmlDocGetRootElement'
topology-xml-libxml.c:(.text+0x99c): undefined reference to `xmlFreeDoc'
topology-xml-libxml.c:(.text+0x9c0): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_backend_exit':
topology-xml-libxml.c:(.text+0x9e3): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_backend_init':
topology-xml-libxml.c:(.text+0xa0e): undefined reference to `xmlCheckVersion'
topology-xml-libxml.c:(.text+0xa3b): undefined reference to `xmlReadFile'
topology-xml-libxml.c:(.text+0xa69): undefined reference to `xmlReadMemory'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc__libxml_export_new_child':
topology-xml-libxml.c:(.text+0xb61): undefined reference to `xmlNewChild'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc__libxml_export_new_prop':
topology-xml-libxml.c:(.text+0xba4): undefined reference to `xmlNewProp'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc__libxml_export_add_content':
topology-xml-libxml.c:(.text+0xbf0): undefined reference to `xmlNodeAddContentLen'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc__libxml2_prepare_export':
topology-xml-libxml.c:(.text+0xc24): undefined reference to `xmlCheckVersion'
topology-xml-libxml.c:(.text+0xc35): undefined reference to `xmlNewDoc'
topology-xml-libxml.c:(.text+0xc4a): undefined reference to `xmlNewNode'
topology-xml-libxml.c:(.text+0xc61): undefined reference to `xmlDocSetRootElement'
topology-xml-libxml.c:(.text+0xc80): undefined reference to `xmlCreateIntSubset'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_export_file':
topology-xml-libxml.c:(.text+0xd26): undefined reference to `xmlSaveFormatFileEnc'
topology-xml-libxml.c:(.text+0xd35): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_export_buffer':
topology-xml-libxml.c:(.text+0xda2): undefined reference to `xmlDocDumpFormatMemoryEnc'
topology-xml-libxml.c:(.text+0xdae): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc__libxml2_prepare_export_diff':
topology-xml-libxml.c:(.text+0xdeb): undefined reference to `xmlCheckVersion'
topology-xml-libxml.c:(.text+0xdfc): undefined reference to `xmlNewDoc'
topology-xml-libxml.c:(.text+0xe11): undefined reference to `xmlNewNode'
topology-xml-libxml.c:(.text+0xe33): undefined reference to `xmlNewProp'
topology-xml-libxml.c:(.text+0xe46): undefined reference to `xmlDocSetRootElement'
topology-xml-libxml.c:(.text+0xe65): undefined reference to `xmlCreateIntSubset'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_export_diff_file':
topology-xml-libxml.c:(.text+0xf07): undefined reference to `xmlSaveFormatFileEnc'
topology-xml-libxml.c:(.text+0xf16): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_export_diff_buffer':
topology-xml-libxml.c:(.text+0xf8e): undefined reference to `xmlDocDumpFormatMemoryEnc'
topology-xml-libxml.c:(.text+0xf9a): undefined reference to `xmlFreeDoc'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-xml-libxml.o): In function `hwloc_libxml_free_buffer':
topology-xml-libxml.c:(.text+0xfb4): undefined reference to `xmlFree'
/global/common/cori/software/hwloc/1.11.4/hsw/lib/libhwloc.a(topology-pci.o): In function `hwloc_look_pci':
topology-pci.c:(.text+0x289): undefined reference to `pci_system_init'
topology-pci.c:(.text+0x2be): undefined reference to `pci_slot_match_iterator_create'
topology-pci.c:(.text+0x2ce): undefined reference to `pci_device_next'
topology-pci.c:(.text+0x2fc): undefined reference to `pci_device_probe'
topology-pci.c:(.text+0x31f): undefined reference to `pci_device_cfg_read'
topology-pci.c:(.text+0x69e): undefined reference to `pci_device_get_vendor_name'
topology-pci.c:(.text+0x6d7): undefined reference to `pci_device_get_device_name'
topology-pci.c:(.text+0x851): undefined reference to `pci_device_next'
topology-pci.c:(.text+0x86c): undefined reference to `pci_iterator_destroy'
topology-pci.c:(.text+0x871): undefined reference to `pci_system_cleanup'
/usr/bin/ld: link errors found, deleting executable `../bin/hpx_runtime'
collect2: error: ld returned 1 exit status
runtime/CMakeFiles/hpx_runtime_exe.dir/build.make:116: recipe for target 'bin/hpx_runtime' failed
make[2]: *** [bin/hpx_runtime] Error 1
make[2]: Leaving directory '/global/u1/j/jkleinh/code/spike/hpx/build'
CMakeFiles/Makefile2:32864: recipe for target 'runtime/CMakeFiles/hpx_runtime_exe.dir/all' failed
make[1]: *** [runtime/CMakeFiles/hpx_runtime_exe.dir/all] Error 2
make[1]: Leaving directory '/global/u1/j/jkleinh/code/spike/hpx/build'
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

I get the same error when using my own build of hwloc unless I follow the above directions to generate a fully static build.

hkaiser commented 7 years ago

@kleinhenz This information should be added to the build instructions of HPX on Cori, thanks!

I assume your initial problem is solved now. Can we now close this ticket?

kleinhenz commented 7 years ago

Yeah, I'm now able to successfully compile statically with mpi and system allocator.