search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.53k stars 435 forks source link

Problems with performance counters in distributed #1415

Closed sithhell closed 9 years ago

sithhell commented 9 years ago

The following counters fail with an error when running in distributed:

The counters are reported to exist. The first fails with:

{stack-trace}: 4 frames:
0x2aaaaaee8590  : hpx::detail::backtrace(unsigned long) + 0x80 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaaaeee87f  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x36f in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaaba798b1  : ??? + 0x2aaaaba798b1 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaab6990d7  : ??? + 0x2aaaab6990d7 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
{env}: 337 entries:
  ALPS_APP_DEPTH=24
  ALPS_APP_ID=11003585
  ALPS_APP_PE=1
  ALTD_ON=1
  ALTD_PATH=/usr/common/usg/altd/2.0
  ALTD_SELECT_ON=0
  ALTD_VERBOSE=0
  ALTD_WORKDIR=/scratch1/altd/logs
  ALT_LINKER=/usr/common/usg/altd/2.0/bin/ld
  ATP_HOME=/opt/cray/atp/1.7.5
  ATP_MRNET_COMM_PATH=/opt/cray/atp/1.7.5/bin/atp_mrnet_commnode_wrapper
  ATP_POST_LINK_OPTS=-Wl,-L/opt/cray/atp/1.7.5/lib/ 
  BASH_ENV=/global/homes/s/sithhell/.bashrc
  BASH_FUNC_module()=() {  eval `/opt/modules/3.2.10.2/bin/modulecmd bash $*`
}
  BOOST_DIR=/usr/common/usg/boost/1.54/intel
  BOOST_LIB=/usr/common/usg/boost/1.54/intel/lib
  BOOST_ROOT=/usr/common/usg/boost/1.54/intel
  COLORTERM=1
  CPU=x86_64
  CRAYOS_VERSION=5.2.40
  CRAYPE_DIR=/opt/cray/craype/2.2.1
  CRAYPE_NETWORK_TARGET=aries
  CRAYPE_VERSION=2.2.1
  CRAY_ALPS_INCLUDE_OPTS=-I/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/include
  CRAY_ALPS_POST_LINK_OPTS=-L/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64
  CRAY_CPU_TARGET=sandybridge
  CRAY_DMAPP_INCLUDE_OPTS=-I/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/include -I/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/include 
  CRAY_DMAPP_POST_LINK_OPTS=-L/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64
  CRAY_GNI_HEADERS_INCLUDE_OPTS=-I/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/include
  CRAY_LD_LIBRARY_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib:/opt/cray/mpt/7.1.1/gni/sma/lib64:/opt/cray/mpt/7.1.1/gni/sma64/lib64:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64:/opt/cray/libsci/13.0.1/INTEL/140/x86_64/lib:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64
  CRAY_LIBSCI_BASE_DIR=/opt/cray/libsci/13.0.1
  CRAY_LIBSCI_DIR=/opt/cray/libsci/13.0.1
  CRAY_LIBSCI_PREFIX_DIR=/opt/cray/libsci/13.0.1/INTEL/140/x86_64
  CRAY_LIBSCI_VERSION=13.0.1
  CRAY_LLM_DIR=/opt/cray/llm/default
  CRAY_MPICH2_BASEDIR=/opt/cray/mpt/7.1.1/gni
  CRAY_MPICH2_DIR=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140
  CRAY_MPICH2_ROOTDIR=/opt/cray/mpt/7.1.1
  CRAY_MPICH2_VER=7.1.1
  CRAY_PMI_INCLUDE_OPTS=-I/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/include
  CRAY_PMI_POST_LINK_OPTS=-L/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64
  CRAY_PORTALS_USE_BLOCKING_POLL=no
  CRAY_PRE_COMPILE_OPTS=-hnetwork=aries
  CRAY_PRGENVINTEL=loaded
  CRAY_SHMEM_BASEDIR=/opt/cray/mpt/7.1.1/gni
  CRAY_SHMEM_DIR=/opt/cray/mpt/7.1.1/gni/sma
  CRAY_SHMEM_ROOTDIR=/opt/cray/mpt/7.1.1
  CRAY_SHMEM_VER=7.1.1
  CRAY_SITE_LIST_DIR=/etc/opt/cray/modules
  CRAY_UDREG_INCLUDE_OPTS=-I/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/include
  CRAY_UDREG_POST_LINK_OPTS=-L/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64
  CRAY_UGNI_INCLUDE_OPTS=-I/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/include
  CRAY_UGNI_POST_LINK_OPTS=-L/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64
  CRAY_XPMEM_INCLUDE_OPTS=-I/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/include
  CRAY_XPMEM_POST_LINK_OPTS=-L/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64
  CSHEDIT=emacs
  CVS_RSH=ssh
  DMAPP_ABORT_ON_ERROR=1
  DVS_MAXNODES=1__
  ENV=/etc/bash.bashrc
  ENVIRONMENT=BATCH
  FFTW_SYSTEM_WISDOM_DIR=/opt/cray/libsci/13.0.1
  FORT_BUFFERED=yes
  FROM_HEADER=
  GCC_PATH=/opt/gcc/4.9.2
  GCC_VERSION=4.9.2
  GNU_VERSION=4.9.2
  GSCRATCH2=/global/scratch2/sd/sithhell
  GSCRATCH=/global/scratch2/sd/sithhell
  G_BROKEN_FILENAMES=1
  G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252
  HISTSIZE=1000
  HOME=/global/homes/s/sithhell
  HOST=nid01666
  HOSTNAME=nid01666
  HOSTTYPE=x86_64
  INCLUDE=/opt/intel/composer_xe_2015.1.133/mkl/include
  INFODIR=/usr/local/info:/usr/share/info:/usr/info
  INFOPATH=/opt/gcc/4.9.2/snos/share/info:/opt/intel/composer_xe_2015.1.133/debugger/gdb/intel64/share/info/:/usr/local/info:/usr/share/info:/usr/info
  INPUTRC=/etc/inputrc
  INTEL_MAJOR_VERSION=15.0
  INTEL_MINOR_VERSION=1.133
  INTEL_PATH=/opt/intel/composer_xe_2015
  INTEL_VERSION=15.0.1.133
  JAVA_BINDIR=/usr/lib64/jvm/java/bin
  JAVA_HOME=/usr/lib64/jvm/java
  JAVA_ROOT=/usr/lib64/jvm/java
  JDK_HOME=/usr/lib64/jvm/java
  JRE_HOME=/usr/lib64/jvm/java/jre
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/usr/common/usg/boost/1.54/intel/lib:/opt/gcc/4.9.2/snos/lib64:/usr/common/usg/darshan/2.3.0/lib:/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64:/opt/intel/composer_xe_2015.1.133/mpirt/lib/intel64:/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64:/usr/syscom/nsg/lib
  LESS=-M -I
  LESSCLOSE=lessclose.sh %s %s
  LESSKEY=/etc/lesskey.bin
  LESSOPEN=lessopen.sh %s
  LESS_ADVANCED_PREPROCESSOR=no
  LIBGL_ALWAYS_INDIRECT=1
  LIBRARYMODULES=/opt/modules/3.2.10.2/init/.librarymodules:acml:alps:cray-dwarf:cray-fftw:cray-ga:cray-hdf5:cray-hdf5-parallel:cray-libsci:cray-libsci_acc:cray-mpich:cray-mpich2:cray-mpich-abi:cray-netcdf:cray-netcdf-hdf5parallel:cray-parallel-netcdf:cray-petsc:cray-petsc-complex:cray-shmem:cray-tpsl:cray-trilinos:cudatoolkit:fftw:ga:hdf5:hdf5-parallel:iobuf:libfast:netcdf:netcdf-hdf5parallel:ntk:onesided:papi:petsc:petsc-complex:pmi:tpsl:trilinos:xt-libsci:xt-mpich2:xt-mpt:xt-papi:/etc/opt/cray/modules/site_librarymodules
  LIBRARY_PATH=/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64
  LIBSCI_BASE_DIR=/opt/cray/libsci/13.0.1
  LIBSCI_VERSION=13.0.1
  LOADEDMODULES=modules/3.2.10.2:nsg/1.2.0:hss-llm/7.2.0:nodestat/2.2-1.0502.53712.3.109.ari:sdb/1.0-1.0502.55976.5.27.ari:alps/5.2.1-2.0502.9072.13.1.ari:lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51:udreg/2.3.2-1.0502.9275.1.12.ari:ugni/5.0-1.0502.9685.4.24.ari:gni-headers/3.0-1.0502.9684.5.2.ari:dmapp/7.0.1-1.0502.9501.5.219.ari:xpmem/0.1-2.0502.55507.3.2.ari:Base-opts/1.0.2-1.0502.53325.1.2.ari:craype-network-aries:craype/2.2.1:intel/15.0.1.133:cray-libsci/13.0.1:pmi/5.0.6-1.0000.10439.140.2.ari:atp/1.7.5:PrgEnv-intel/5.2.40:craype-ivybridge:cray-shmem/7.1.1:cray-mpich/7.1.1:torque/5.0.1:moab/8.0.1-2014110616-5c7a394-sles11:altd/2.0:darshan/2.3.0:usg-default-modules/1.1:gcc/4.9.2:boost/1.54
  LOGNAME=sithhell
  LS_COLORS=
  LS_OPTIONS=-N --color=none -T 0
  MACHTYPE=x86_64-suse-linux
  MAIL=/var/spool/mail/sithhell
  MANPATH=/opt/gcc/4.9.2/snos/share/man:/usr/common/usg/man:/usr/common/mss/man:/usr/common/nsg/man:/opt/moab/8.0.1/share/man/:/opt/torque/5.0.1/share/man:/opt/cray/mpt/7.1.1/gni/man/mpich2:/opt/cray/mpt/7.1.1/gni/man/shmem:/opt/cray/atp/1.7.5/man:/opt/cray/libsci/13.0.1/man:/opt/intel/composer_xe_2015.1.133/man/en_US:/opt/cray/craype/2.2.1/man:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/man:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/man:/opt/cray/llm/default/man:/usr/syscom/nsg/man:/opt/modules/3.2.10.2/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/man:/opt/cray/share/man
  MAN_POSIXLY_CORRECT=1
  MINICOM=-c on
  MKLROOT=/opt/intel/composer_xe_2015.1.133/mkl
  MOABHOMEDIR=/var/spool/moab
  MODULEPATH=/opt/cray/craype/default/modulefiles:/opt/cray/ari/modulefiles:/opt/cray/modulefiles:/opt/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/syscom/nsg/opt/modulefiles:/usr/common/das/Modules/modulefiles:/usr/common/ftg/Modules/modulefiles:/usr/common/graphics/Modules/modulefiles:/usr/common/jgi/Modules/modulefiles:/usr/common/tig/Modules/modulefiles
  MODULESHOME=/opt/modules/3.2.10.2
  MODULE_VERSION=3.2.10.2
  MODULE_VERSION_STACK=3.2.10.2
  MORE=-sl
  MPICH_ABORT_ON_ERROR=1
  MPICH_DIR=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140
  MPICH_MPIIO_DVS_MAXNODES=14
  NERSC_HOST=edison
  NLSPATH=/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64/locale/%l_%t/%N
  NNTPSERVER=news
  NSG_HOME=/usr/syscom/nsg
  OLDPWD=/global/u2/s/sithhell
  OSTYPE=linux
  PAGER=less
  PATH=/opt/gcc/4.9.2/bin:/usr/common/usg/darshan/2.3.0/bin:/usr/common/usg/altd/2.0/bin:/usr/common/usg/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/moab/8.0.1/bin:/opt/moab/8.0.1/sbin:/opt/torque/5.0.1/sbin:/opt/torque/5.0.1/bin:/opt/cray/mpt/7.1.1/gni/bin:/opt/cray/atp/1.7.5/bin:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/bin:/opt/intel/composer_xe_2015.1.133/bin/intel64:/opt/cray/craype/2.2.1/bin:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/bin:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/bin:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/bin:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/bin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/sbin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/bin:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/sbin:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/bin:/opt/cray/sdb/1.0-1.0502.55976.5.27.ari/bin:/opt/cray/nodestat/2.2-1.0502.53712.3.109.ari/bin:/opt/cray/llm/default/bin:/opt/cray/llm/default/etc:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/modules/3.2.10.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/opt/cray/bin
  PBSCOREDUMP=True
  PBS_ENVIRONMENT=PBS_BATCH
  PBS_GPUFILE=/var/spool/torque/aux//2552355.edique02gpu
  PBS_JOBCOOKIE=CDE8F2D5FA48349097EF86615F835CB9
  PBS_JOBID=2552355.edique02
  PBS_JOBNAME=nbody_weak_2
  PBS_MICFILE=/var/spool/torque/aux//2552355.edique02mic
  PBS_MOMPORT=15003
  PBS_NODEFILE=/var/spool/torque/aux//2552355.edique02
  PBS_NODENUM=0
  PBS_NP=48
  PBS_NUM_NODES=1
  PBS_NUM_PPN=1
  PBS_O_HOME=/global/homes/s/sithhell
  PBS_O_HOST=edison10
  PBS_O_LANG=en_US.UTF-8
  PBS_O_LOGNAME=sithhell
  PBS_O_MAIL=/var/mail/sithhell
  PBS_O_PATH=/usr/common/usg/darshan/2.3.0/bin:/usr/common/usg/altd/2.0/bin:/usr/common/usg/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/moab/8.0.1/bin:/opt/moab/8.0.1/sbin:/opt/torque/5.0.1/sbin:/opt/torque/5.0.1/bin:/opt/cray/mpt/7.1.1/gni/bin:/opt/cray/atp/1.7.5/bin:/opt/cray/rca/1.0.0-2.0502.53711.3.127.ari/bin:/opt/cray/alps/5.2.1-2.0502.9041.11.6.ari/sbin:/opt/cray/alps/5.2.1-2.0502.9041.11.6.ari/bin:/opt/cray/dvs/2.5_0.9.0-1.0502.1873.1.145.ari/bin:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/bin:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/bin:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/bin:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/bin:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/bin:/opt/intel/composer_xe_2015.1.133/bin/intel64:/opt/cray/craype/2.2.1/bin:/opt/cray/switch/1.0-1.0502.54233.2.96.ari/bin:/opt/cray/eslogin/eswrap/1.1.0-1.020200.1130.0/bin:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/modules/3.2.10.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/opt/cray/bin
  PBS_O_QUEUE=regular
  PBS_O_SERVER=edique02
  PBS_O_SHELL=/bin/bash
  PBS_O_SUBMIT_FILTER=/usr/syscom/nsg/sbin/submit_filter
  PBS_O_WORKDIR=/global/project/projectdirs/m1881/hpx/nbody_demo/edison/runs
  PBS_QUEUE=reg_small
  PBS_TASKNUM=1
  PBS_VERSION=TORQUE-5.0.1
  PBS_VNODENUM=0
  PBS_WALLTIME=1800
  PELOCAL_PRGENV=true
  PE_CRAY_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/CRAY/83/lib/pkgconfig:/opt/cray/hdf5-parallel/1.8.13/CRAY/83/lib/pkgconfig:/opt/cray/netcdf/4.3.2/CRAY/83/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.2/CRAY/83/lib/pkgconfig:/opt/cray/parallel-netcdf/1.5.0/CRAY/83/lib/pkgconfig:/opt/cray/hdf5/1.8.13/CRAY/83/lib/pkgconfig
  PE_CXX_PKGCONFIG_LIBS=mpichcxx
  PE_ENV=INTEL
  PE_FFTW_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_FFTW_DEFAULT_TARGET_haswell=haswell
  PE_FFTW_DEFAULT_TARGET_interlagos=interlagos
  PE_FFTW_DEFAULT_TARGET_sandybridge=sandybridge
  PE_FFTW_DEFAULT_TARGET_x86_64=x86_64
  PE_FFTW_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/fftw/3.3.4.1/@PE_FFTW_DEFAULT_TARGET@/lib/pkgconfig
  PE_FORTRAN_PKGCONFIG_LIBS=mpichf90
  PE_GA_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_GA_DEFAULT_GENCOMPS_GNU=49 48
  PE_GA_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/@PRGENV@/@PE_GA_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_GA_DEFAULT_VOLATILE_PRGENV=GNU
  PE_HDF5_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_HDF5_DEFAULT_GENCOMPS_GNU=49 48
  PE_HDF5_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5/1.8.13/@PRGENV@/@PE_HDF5_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_HDF5_DEFAULT_VOLATILE_PRGENV=GNU
  PE_HDF5_PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_HDF5_PARALLEL_DEFAULT_GENCOMPS_GNU=49 48
  PE_HDF5_PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5-parallel/1.8.13/@PRGENV@/@PE_HDF5_PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU
  PE_INTEL_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/INTEL/140/lib/pkgconfig:/opt/cray/hdf5-parallel/1.8.13/INTEL/140/lib/pkgconfig:/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib/pkgconfig:/opt/cray/netcdf/4.3.2/INTEL/140/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib/pkgconfig:/opt/cray/parallel-netcdf/1.5.0/INTEL/140/lib/pkgconfig:/opt/cray/hdf5/1.8.13/INTEL/140/lib/pkgconfig
  PE_INTEL_FIXED_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib/pkgconfig
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_interlagos=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_LIBSCI_DEFAULT_OMP_REQUIRES_openmp=_mp
  PE_LIBSCI_DEFAULT_PKGCONFIG_VARIABLES=PE_LIBSCI_DEFAULT_OMP_REQUIRES_@openmp@
  PE_LIBSCI_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_LIBSCI_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/13.0.1/@PRGENV@/@PE_LIBSCI_DEFAULT_GENCOMPS@/@PE_LIBSCI_DEFAULT_TARGET@/lib/pkgconfig
  PE_LIBSCI_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_LIBSCI_GENCOMPS_CRAY_haswell=83
  PE_LIBSCI_GENCOMPS_CRAY_interlagos=83
  PE_LIBSCI_GENCOMPS_CRAY_sandybridge=83
  PE_LIBSCI_GENCOMPS_CRAY_x86_64=83
  PE_LIBSCI_GENCOMPS_GNU_haswell=49 48
  PE_LIBSCI_GENCOMPS_GNU_interlagos=49 48
  PE_LIBSCI_GENCOMPS_GNU_sandybridge=49 48
  PE_LIBSCI_GENCOMPS_GNU_x86_64=49 48
  PE_LIBSCI_GENCOMPS_INTEL_haswell=140
  PE_LIBSCI_GENCOMPS_INTEL_interlagos=140
  PE_LIBSCI_GENCOMPS_INTEL_sandybridge=140
  PE_LIBSCI_GENCOMPS_INTEL_x86_64=140
  PE_LIBSCI_MODULE_NAME=cray-libsci/13.0.1
  PE_LIBSCI_OMP_REQUIRES_openmp=_mp
  PE_LIBSCI_PKGCONFIG_LIBS=libsci_mpi:libsci
  PE_LIBSCI_PKGCONFIG_VARIABLES=PE_LIBSCI_OMP_REQUIRES_@openmp@
  PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH
  PE_LIBSCI_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/13.0.1/@PRGENV@/@PE_LIBSCI_GENCOMPS@/@PE_LIBSCI_TARGET@/lib/pkgconfig
  PE_LIBSCI_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_MPICH_DEFAULT_DIR_CRAY_DEFAULT64=64
  PE_MPICH_DEFAULT_FIXED_PRGENV=INTEL
  PE_MPICH_DEFAULT_GENCOMPS_CRAY=83
  PE_MPICH_DEFAULT_GENCOMPS_GNU=49 48
  PE_MPICH_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-@PRGENV@@PE_MPICH_DEFAULT_DIR_DEFAULT64@/@PE_MPICH_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_MPICH_DEFAULT_VOLATILE_PRGENV=CRAY GNU
  PE_MPICH_DIR_CRAY_DEFAULT64=64
  PE_MPICH_FIXED_PRGENV=INTEL
  PE_MPICH_GENCOMPS_CRAY=83
  PE_MPICH_GENCOMPS_GNU=49 48
  PE_MPICH_MODULE_NAME=cray-mpich
  PE_MPICH_MULTITHREADED_LIBS_multithreaded=_mt
  PE_MPICH_NV_LIBS_nvidia20=-lcudart
  PE_MPICH_NV_LIBS_nvidia35=-lcudart
  PE_MPICH_PKGCONFIG_VARIABLES=PE_MPICH_NV_LIBS_@accelerator@:PE_MPICH_MULTITHREADED_LIBS_@multithreaded@
  PE_MPICH_TARGET_VAR_nvidia20=-lcudart
  PE_MPICH_TARGET_VAR_nvidia35=-lcudart
  PE_MPICH_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-@PRGENV@@PE_MPICH_DIR_DEFAULT64@/@PE_MPICH_GENCOMPS@/lib/pkgconfig
  PE_MPICH_VOLATILE_PRGENV=CRAY GNU
  PE_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_NETCDF_DEFAULT_GENCOMPS_GNU=49 48
  PE_NETCDF_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5
  PE_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf/4.3.2/@PRGENV@/@PE_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU
  PE_NETCDF_HDF5PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS_GNU=49 48
  PE_NETCDF_HDF5PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5_PARALLEL PE_MPICH
  PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf-hdf5parallel/4.3.2/@PRGENV@/@PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU
  PE_PAPI_DEFAULT_ACCEL_LIBS_nvidia35=,-lcupti,-lcudart,-lcuda
  PE_PAPI_DEFAULT_PKGCONFIG_VARIABLES=PE_PAPI_ACCEL_LIBS_@accelerator@
  PE_PAPI_DEFAULT_TARGET_VAR_nvidia35=,-lcupti,-lcudart,-lcuda
  PE_PARALLEL_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS_GNU=49 48
  PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/parallel-netcdf/1.5.0/@PRGENV@/@PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_PETSC_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_PETSC_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI:PE_TPSL
  PE_PETSC_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/petsc/3.5.2.1/complex/@PRGENV@/@PE_PETSC_DEFAULT_GENCOMPS@/@PE_PETSC_DEFAULT_TARGET@/lib/pkgconfig
  PE_PETSC_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_PKGCONFIG_DEFAULT_PRODUCTS=PE_LIBSCI:PE_GA:PE_HDF5_PARALLEL:PE_TPSL:PE_MPICH:PE_NETCDF:PE_PETSC:PE_TRILINOS:PE_NETCDF_HDF5PARALLEL:PE_FFTW:PE_PARALLEL_NETCDF:PE_HDF5
  PE_PKGCONFIG_LIBS=darshan-runtime:mpich:sma:AtpSigHandler:libsci_mpi:libsci
  PE_PKGCONFIG_PRODUCTS=PE_MPICH:PE_SMA:PE_LIBSCI
  PE_PKGCONFIG_PRODUCTS_DEFAULT=PE_PAPI
  PE_PRODUCT_LIST=CRAYPE_IVYBRIDGE:CRAY_PMI:CRAY_LIBSCI:CRAYPE:INTEL:CRAY_XPMEM:CRAY_DMAPP:CRAY_UGNI:CRAY_UDREG:CRAY_ALPS:CRAY_LLM
  PE_SMA_DIR_CRAY_DEFAULT64=64
  PE_SMA_DIR_PGI_DEFAULT64=64
  PE_SMA_MODULE_NAME=cray-shmem
  PE_SMA_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/sma@PE_SMA_DIR_DEFAULT64@/lib64/pkgconfig
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_TPSL_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_interlagos=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_TPSL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI
  PE_TPSL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/tpsl/1.4.3/@PRGENV@/@PE_TPSL_DEFAULT_GENCOMPS@/@PE_TPSL_DEFAULT_TARGET@/lib/pkgconfig
  PE_TPSL_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_TRILINOS_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_TRILINOS_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_TRILINOS_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_TRILINOS_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_LIBSCI:PE_TPSL
  PE_TRILINOS_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/trilinos/11.12.1.0/@PRGENV@/@PE_TRILINOS_DEFAULT_GENCOMPS@/@PE_TRILINOS_DEFAULT_TARGET@/lib/pkgconfig
  PE_TRILINOS_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PKGCONFIG_ENABLED=1
  PKG_CONFIG_PATH=/usr/common/usg/darshan/2.3.0/lib/pkgconfig:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64/pkgconfig:/opt/cray/craype/2.2.1/pkg-config:/opt/cray/iobuf/2.0.5/lib/pkgconfig:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64/pkgconfig:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64/pkgconfig:/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/lib64/pkgconfig:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64/pkgconfig:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64/pkgconfig:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64/pkgconfig:/opt/cray/atp/1.7.5/lib/pkgconfig
  PKG_CONFIG_PATH_DEFAULT=/opt/cray/papi/5.3.2.1/lib64/pkgconfig
  PMI_FORK_RANK=1
  PMI_GNI_COOKIE=1768292352:1768357888
  PMI_GNI_DEV_ID=0
  PMI_GNI_LOC_ADDR=7742:7742
  PMI_GNI_PTAG=44:45
  PRGENVMODULES=/opt/modules/3.2.10.2/init/.prgenvmodules:PrgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi
  PROFILEREAD=true
  PWD=/global/homes/s/sithhell/project/hpx/edison/nbody_demo-release
  PYTHONPATH=/usr/common/usg/altd/2.0/lib64/python2.6/site-packages:/opt/cray/sdb/1.0-1.0502.55976.5.27.ari/lib64/py
  PYTHONSTARTUP=/etc/pythonstart
  QTDIR=/usr/lib/qt3
  QT_SYSTEM_DIR=/usr/share/desktop-data
  RCLOCAL_BASEOPTS=true
  RCLOCAL_PRGENV=true
  SCRATCH=/scratch2/scratchdirs/sithhell
  SDK_HOME=/usr/lib64/jvm/java
  SHELL=/bin/bash
  SHLVL=2
  SHMEM_ABORT_ON_ERROR=1
  TARGETMODULES=/opt/modules/3.2.10.2/init/.targetmodules:craype-abudhabi:craype-abudhabi-cu:craype-accel-host:craype-accel-nvidia20:craype-accel-nvidia30:craype-accel-nvidia35:craype-barcelona:craype-broadwell:craype-haswell:craype-hugepages128K:craype-hugepages128M:craype-hugepages16M:craype-hugepages256M:craype-hugepages2M:craype-hugepages32M:craype-hugepages4M:craype-hugepages512K:craype-hugepages512M:craype-hugepages64M:craype-hugepages8M:craype-intel-knc:craype-interlagos:craype-interlagos-cu:craype-istanbul:craype-ivybridge:craype-mc12:craype-mc8:craype-mic-knl:craype-network-aries:craype-network-gemini:craype-network-infiniband:craype-network-none:craype-network-seastar:craype-sandybridge:craype-shanghai:craype-target-compute_node:craype-target-local_host:craype-target-native:craype-xeon:xtpe-barcelona:xtpe-interlagos:xtpe-interlagos-cu:xtpe-istanbul:xtpe-mc12:xtpe-mc8:xtpe-network-gemini:xtpe-network-seastar:xtpe-shanghai:xtpe-target-native:xtpe-xeon:/etc/opt/cray/modules/site_targetmodules
  TMPDIR=/scratch2/scratchdirs/sithhell
  TOOLMODULES=/opt/modules/3.2.10.2/init/.toolmodules:apprentice:apprentice2:atp:chapel:cray-lgdb:craypat:craypkg-gen:cray-snplauncher:ddt:gdb:iobuf:papi:perftools:perftools-lite:stat:totalview:xt-craypat:xt-lgdb:xt-papi:xt-totalview:/etc/opt/cray/modules/site_toolmodules
  USER=sithhell
  USERMODULES=/opt/modules/3.2.10.2/init/.usermodules:acml:alps:apprentice:apprentice2:atp:blcr:cce:chapel:cray-ccdb:cray-fftw:cray-ga:cray-hdf5:cray-hdf5-parallel:cray-lgdb:cray-libsci:cray-libsci_acc:cray-mpich:cray-mpich2:cray-mpich-compat:cray-netcdf:cray-netcdf-hdf5parallel:cray-parallel-netcdf:craypat:craype:cray-petsc:cray-petsc-complex:craypkg-gen:cray-shmem:cray-snplauncher:cray-tpsl:cray-trilinos:cudatoolkit:ddt:fftw:ga:gcc:hdf5:hdf5-parallel:intel:iobuf:java:lgdb:libfast:libsci_acc:mpich1:netcdf:netcdf-hdf5parallel:netcdf-nofsync:netcdf-nofsync-hdf5parallel:ntk:onesided:papi:parallel-netcdf:pathscale:perftools:perftools-lite:petsc:petsc-complex:pgi:pmi:PrgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi:stat:totalview:tpsl:trilinos:xt-asyncpe:xt-craypat:xt-lgdb:xt-libsci:xt-mpich2:xt-mpt:xt-papi:xt-shmem:xt-totalview:/etc/opt/cray/modules/site_usermodules
  WINDOWMANAGER=
  XCURSOR_THEME=crystalwhite
  XDG_CONFIG_DIRS=/etc/xdg
  XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share
  XKEYSYMDB=/usr/share/X11/XKeysymDB
  XNLSPATH=/usr/share/X11/nls
  XTOS_VERSION=5.2.40
  XTPE_NETWORK_TARGET=aries
  _=/usr/common/usg/altd/2.0/bin/aprun
  _DMAPPI_RESERVED_MDDS=1
  _LMFILES_=/opt/modulefiles/modules/3.2.10.2:/usr/syscom/nsg/modulefiles/nsg/1.2.0:/opt/modulefiles/hss-llm/7.2.0:/opt/cray/ari/modulefiles/nodestat/2.2-1.0502.53712.3.109.ari:/opt/cray/ari/modulefiles/sdb/1.0-1.0502.55976.5.27.ari:/opt/cray/ari/modulefiles/alps/5.2.1-2.0502.9072.13.1.ari:/opt/cray/modulefiles/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51:/opt/cray/ari/modulefiles/udreg/2.3.2-1.0502.9275.1.12.ari:/opt/cray/ari/modulefiles/ugni/5.0-1.0502.9685.4.24.ari:/opt/cray/ari/modulefiles/gni-headers/3.0-1.0502.9684.5.2.ari:/opt/cray/ari/modulefiles/dmapp/7.0.1-1.0502.9501.5.219.ari:/opt/cray/ari/modulefiles/xpmem/0.1-2.0502.55507.3.2.ari:/opt/modulefiles/Base-opts/1.0.2-1.0502.53325.1.2.ari:/opt/cray/craype/default/modulefiles/craype-network-aries:/opt/cray/modulefiles/craype/2.2.1:/opt/modulefiles/intel/15.0.1.133:/opt/cray/modulefiles/cray-libsci/13.0.1:/opt/cray/ari/modulefiles/pmi/5.0.6-1.0000.10439.140.2.ari:/opt/cray/modulefiles/atp/1.7.5:/opt/cray/modulefiles/PrgEnv-intel/5.2.40:/opt/cray/craype/default/modulefiles/craype-ivybridge:/opt/cray/modulefiles/cray-shmem/7.1.1:/opt/cray/modulefiles/cray-mpich/7.1.1:/opt/modulefiles/torque/5.0.1:/opt/modulefiles/moab/8.0.1-2014110616-5c7a394-sles11:/usr/common/usg/Modules/modulefiles/altd/2.0:/usr/common/usg/Modules/modulefiles/darshan/2.3.0:/usr/common/usg/Modules/modulefiles/usg-default-modules/1.1:/opt/modulefiles/gcc/4.9.2:/usr/common/usg/Modules/modulefiles/boost/1.54
{locality-id}: 1
{hostname}: [ (mpi:1) ]
{process-id}: 2278
{function}: create_counter_local
{file}: /project/projectdirs/m1881/hpx/source/src/performance_counters/counters.cpp
{line}: 680
{os-thread}: 10, worker-thread#6
{thread-id}: 00002aaac50297a0
{thread-description}: <unknown>
{state}: startup
{auxinfo}: 
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_PARCELPORT_TCP=OFF
  HPX_PARCELPORT_MPI=ON (MPICH V3.1.2, MPI V3.0)
  HPX_PARCELPORT_IPC=OFF
  HPX_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_RUN_MAIN_EVERYWHERE=OFF
  HPX_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX (configured)=/global/homes/s/sithhell/project/hpx/edison/hpx-release
  HPX_PREFIX=/global/homes/s/sithhell/project/hpx/edison/hpx-release
{version}: V0.9.10-rc1 (AGAS: V3.0), Git: 17b2f0c4d88a939359ee6d521694fd6ba00d72c4
{boost}: V1.56.0
{build-type}: release
{date}: Mar 16 2015 12:12:46
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20141030
{what}: no create function for performance counter found: /agas{locality#1/total}/component/count (counter type is not defined: HPX(bad_parameter)): HPX(bad_parameter):

The second with:

{stack-trace}: 4 frames:
0x2aaaaaee8590  : hpx::detail::backtrace(unsigned long) + 0x80 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaaaeee87f  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x36f in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaaba798b1  : ??? + 0x2aaaaba798b1 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
0x2aaaab6990d7  : ??? + 0x2aaaab6990d7 in /global/homes/s/sithhell/project/hpx/edison/hpx-release/lib/libhpx.so.0
{env}: 337 entries:
  ALPS_APP_DEPTH=24
  ALPS_APP_ID=11003791
  ALPS_APP_PE=1
  ALTD_ON=1
  ALTD_PATH=/usr/common/usg/altd/2.0
  ALTD_SELECT_ON=0
  ALTD_VERBOSE=0
  ALTD_WORKDIR=/scratch1/altd/logs
  ALT_LINKER=/usr/common/usg/altd/2.0/bin/ld
  ATP_HOME=/opt/cray/atp/1.7.5
  ATP_MRNET_COMM_PATH=/opt/cray/atp/1.7.5/bin/atp_mrnet_commnode_wrapper
  ATP_POST_LINK_OPTS=-Wl,-L/opt/cray/atp/1.7.5/lib/ 
  BASH_ENV=/global/homes/s/sithhell/.bashrc
  BASH_FUNC_module()=() {  eval `/opt/modules/3.2.10.2/bin/modulecmd bash $*`
}
  BOOST_DIR=/usr/common/usg/boost/1.54/intel
  BOOST_LIB=/usr/common/usg/boost/1.54/intel/lib
  BOOST_ROOT=/usr/common/usg/boost/1.54/intel
  COLORTERM=1
  CPU=x86_64
  CRAYOS_VERSION=5.2.40
  CRAYPE_DIR=/opt/cray/craype/2.2.1
  CRAYPE_NETWORK_TARGET=aries
  CRAYPE_VERSION=2.2.1
  CRAY_ALPS_INCLUDE_OPTS=-I/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/include
  CRAY_ALPS_POST_LINK_OPTS=-L/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64
  CRAY_CPU_TARGET=sandybridge
  CRAY_DMAPP_INCLUDE_OPTS=-I/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/include -I/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/include 
  CRAY_DMAPP_POST_LINK_OPTS=-L/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64
  CRAY_GNI_HEADERS_INCLUDE_OPTS=-I/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/include
  CRAY_LD_LIBRARY_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib:/opt/cray/mpt/7.1.1/gni/sma/lib64:/opt/cray/mpt/7.1.1/gni/sma64/lib64:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64:/opt/cray/libsci/13.0.1/INTEL/140/x86_64/lib:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64
  CRAY_LIBSCI_BASE_DIR=/opt/cray/libsci/13.0.1
  CRAY_LIBSCI_DIR=/opt/cray/libsci/13.0.1
  CRAY_LIBSCI_PREFIX_DIR=/opt/cray/libsci/13.0.1/INTEL/140/x86_64
  CRAY_LIBSCI_VERSION=13.0.1
  CRAY_LLM_DIR=/opt/cray/llm/default
  CRAY_MPICH2_BASEDIR=/opt/cray/mpt/7.1.1/gni
  CRAY_MPICH2_DIR=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140
  CRAY_MPICH2_ROOTDIR=/opt/cray/mpt/7.1.1
  CRAY_MPICH2_VER=7.1.1
  CRAY_PMI_INCLUDE_OPTS=-I/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/include
  CRAY_PMI_POST_LINK_OPTS=-L/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64
  CRAY_PORTALS_USE_BLOCKING_POLL=no
  CRAY_PRE_COMPILE_OPTS=-hnetwork=aries
  CRAY_PRGENVINTEL=loaded
  CRAY_SHMEM_BASEDIR=/opt/cray/mpt/7.1.1/gni
  CRAY_SHMEM_DIR=/opt/cray/mpt/7.1.1/gni/sma
  CRAY_SHMEM_ROOTDIR=/opt/cray/mpt/7.1.1
  CRAY_SHMEM_VER=7.1.1
  CRAY_SITE_LIST_DIR=/etc/opt/cray/modules
  CRAY_UDREG_INCLUDE_OPTS=-I/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/include
  CRAY_UDREG_POST_LINK_OPTS=-L/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64
  CRAY_UGNI_INCLUDE_OPTS=-I/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/include
  CRAY_UGNI_POST_LINK_OPTS=-L/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64
  CRAY_XPMEM_INCLUDE_OPTS=-I/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/include
  CRAY_XPMEM_POST_LINK_OPTS=-L/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64
  CSHEDIT=emacs
  CVS_RSH=ssh
  DMAPP_ABORT_ON_ERROR=1
  DVS_MAXNODES=1__
  ENV=/etc/bash.bashrc
  FFTW_SYSTEM_WISDOM_DIR=/opt/cray/libsci/13.0.1
  FORT_BUFFERED=yes
  FROM_HEADER=
  GCC_PATH=/opt/gcc/4.9.2
  GCC_VERSION=4.9.2
  GNU_VERSION=4.9.2
  GSCRATCH2=/global/scratch2/sd/sithhell
  GSCRATCH=/global/scratch2/sd/sithhell
  G_BROKEN_FILENAMES=1
  G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252
  HISTSIZE=1000
  HOME=/global/homes/s/sithhell
  HOST=nid01665
  HOSTNAME=nid01665
  HOSTTYPE=x86_64
  INCLUDE=/opt/intel/composer_xe_2015.1.133/mkl/include
  INFODIR=/usr/local/info:/usr/share/info:/usr/info
  INFOPATH=/opt/gcc/4.9.2/snos/share/info:/opt/intel/composer_xe_2015.1.133/debugger/gdb/intel64/share/info/:/usr/local/info:/usr/share/info:/usr/info
  INPUTRC=/etc/inputrc
  INTEL_MAJOR_VERSION=15.0
  INTEL_MINOR_VERSION=1.133
  INTEL_PATH=/opt/intel/composer_xe_2015
  INTEL_VERSION=15.0.1.133
  JAVA_BINDIR=/usr/lib64/jvm/java/bin
  JAVA_HOME=/usr/lib64/jvm/java
  JAVA_ROOT=/usr/lib64/jvm/java
  JDK_HOME=/usr/lib64/jvm/java
  JRE_HOME=/usr/lib64/jvm/java/jre
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/usr/common/usg/boost/1.54/intel/lib:/opt/gcc/4.9.2/snos/lib64:/usr/common/usg/darshan/2.3.0/lib:/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64:/opt/intel/composer_xe_2015.1.133/mpirt/lib/intel64:/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64:/usr/syscom/nsg/lib
  LESS=-M -I
  LESSCLOSE=lessclose.sh %s %s
  LESSKEY=/etc/lesskey.bin
  LESSOPEN=lessopen.sh %s
  LESS_ADVANCED_PREPROCESSOR=no
  LIBGL_ALWAYS_INDIRECT=1
  LIBRARYMODULES=/opt/modules/3.2.10.2/init/.librarymodules:acml:alps:cray-dwarf:cray-fftw:cray-ga:cray-hdf5:cray-hdf5-parallel:cray-libsci:cray-libsci_acc:cray-mpich:cray-mpich2:cray-mpich-abi:cray-netcdf:cray-netcdf-hdf5parallel:cray-parallel-netcdf:cray-petsc:cray-petsc-complex:cray-shmem:cray-tpsl:cray-trilinos:cudatoolkit:fftw:ga:hdf5:hdf5-parallel:iobuf:libfast:netcdf:netcdf-hdf5parallel:ntk:onesided:papi:petsc:petsc-complex:pmi:tpsl:trilinos:xt-libsci:xt-mpich2:xt-mpt:xt-papi:/etc/opt/cray/modules/site_librarymodules
  LIBRARY_PATH=/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64
  LIBSCI_BASE_DIR=/opt/cray/libsci/13.0.1
  LIBSCI_VERSION=13.0.1
  LOADEDMODULES=modules/3.2.10.2:nsg/1.2.0:hss-llm/7.2.0:nodestat/2.2-1.0502.53712.3.109.ari:sdb/1.0-1.0502.55976.5.27.ari:alps/5.2.1-2.0502.9072.13.1.ari:lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51:udreg/2.3.2-1.0502.9275.1.12.ari:ugni/5.0-1.0502.9685.4.24.ari:gni-headers/3.0-1.0502.9684.5.2.ari:dmapp/7.0.1-1.0502.9501.5.219.ari:xpmem/0.1-2.0502.55507.3.2.ari:Base-opts/1.0.2-1.0502.53325.1.2.ari:craype-network-aries:craype/2.2.1:intel/15.0.1.133:cray-libsci/13.0.1:pmi/5.0.6-1.0000.10439.140.2.ari:atp/1.7.5:PrgEnv-intel/5.2.40:craype-ivybridge:cray-shmem/7.1.1:cray-mpich/7.1.1:torque/5.0.1:moab/8.0.1-2014110616-5c7a394-sles11:altd/2.0:darshan/2.3.0:usg-default-modules/1.1:gcc/4.9.2:boost/1.54
  LOGNAME=sithhell
  LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.avi=01;35:*.bmp=01;35:*.fli=01;35:*.gif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mng=01;35:*.mov=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.tga=01;35:*.tif=01;35:*.xbm=01;35:*.xpm=01;35:*.dl=01;35:*.gl=01;35:*.wmv=01;35:*.aiff=00;32:*.au=00;32:*.mid=00;32:*.mp3=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:
  LS_OPTIONS=-N --color=none -T 0
  MACHTYPE=x86_64-suse-linux
  MAIL=/var/spool/mail/sithhell
  MANPATH=/opt/gcc/4.9.2/snos/share/man:/usr/common/usg/man:/usr/common/mss/man:/usr/common/nsg/man:/opt/moab/8.0.1/share/man/:/opt/torque/5.0.1/share/man:/opt/cray/mpt/7.1.1/gni/man/mpich2:/opt/cray/mpt/7.1.1/gni/man/shmem:/opt/cray/atp/1.7.5/man:/opt/cray/libsci/13.0.1/man:/opt/intel/composer_xe_2015.1.133/man/en_US:/opt/cray/craype/2.2.1/man:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/man:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/man:/opt/cray/llm/default/man:/usr/syscom/nsg/man:/opt/modules/3.2.10.2/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/man:/opt/cray/share/man
  MAN_POSIXLY_CORRECT=1
  MINICOM=-c on
  MKLROOT=/opt/intel/composer_xe_2015.1.133/mkl
  MOABHOMEDIR=/var/spool/moab
  MODULEPATH=/opt/cray/craype/default/modulefiles:/opt/cray/ari/modulefiles:/opt/cray/modulefiles:/opt/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/syscom/nsg/opt/modulefiles:/usr/common/das/Modules/modulefiles:/usr/common/ftg/Modules/modulefiles:/usr/common/graphics/Modules/modulefiles:/usr/common/jgi/Modules/modulefiles:/usr/common/tig/Modules/modulefiles
  MODULESHOME=/opt/modules/3.2.10.2
  MODULE_VERSION=3.2.10.2
  MODULE_VERSION_STACK=3.2.10.2
  MORE=-sl
  MPICH_ABORT_ON_ERROR=1
  MPICH_DIR=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140
  MPICH_MPIIO_DVS_MAXNODES=14
  NERSC_HOST=edison
  NLSPATH=/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64/locale/%l_%t/%N
  NNTPSERVER=news
  NSG_HOME=/usr/syscom/nsg
  OLDPWD=/global/homes/s/sithhell
  OSTYPE=linux
  PAGER=less
  PATH=/opt/gcc/4.9.2/bin:/usr/common/usg/darshan/2.3.0/bin:/usr/common/usg/altd/2.0/bin:/usr/common/usg/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/moab/8.0.1/bin:/opt/moab/8.0.1/sbin:/opt/torque/5.0.1/sbin:/opt/torque/5.0.1/bin:/opt/cray/mpt/7.1.1/gni/bin:/opt/cray/atp/1.7.5/bin:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/bin:/opt/intel/composer_xe_2015.1.133/bin/intel64:/opt/cray/craype/2.2.1/bin:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/bin:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/bin:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/bin:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/bin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/sbin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51/bin:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/sbin:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/bin:/opt/cray/sdb/1.0-1.0502.55976.5.27.ari/bin:/opt/cray/nodestat/2.2-1.0502.53712.3.109.ari/bin:/opt/cray/llm/default/bin:/opt/cray/llm/default/etc:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/modules/3.2.10.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/opt/cray/bin
  PBSCOREDUMP=True
  PBS_ENVIRONMENT=PBS_INTERACTIVE
  PBS_GPUFILE=/var/spool/torque/aux//2552368.edique02gpu
  PBS_JOBCOOKIE=3E88246E84E7B5A10FEE304BC79B1FF8
  PBS_JOBID=2552368.edique02
  PBS_JOBNAME=STDIN
  PBS_MICFILE=/var/spool/torque/aux//2552368.edique02mic
  PBS_MOMPORT=15003
  PBS_NODEFILE=/var/spool/torque/aux//2552368.edique02
  PBS_NODENUM=0
  PBS_NP=48
  PBS_NUM_NODES=1
  PBS_NUM_PPN=1
  PBS_O_HOME=/global/homes/s/sithhell
  PBS_O_HOST=edison10
  PBS_O_LANG=en_US.UTF-8
  PBS_O_LOGNAME=sithhell
  PBS_O_MAIL=/var/mail/sithhell
  PBS_O_PATH=/opt/gcc/4.9.2/bin:/usr/common/usg/altd/2.0/bin:/usr/common/usg/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/moab/8.0.1/bin:/opt/moab/8.0.1/sbin:/opt/torque/5.0.1/sbin:/opt/torque/5.0.1/bin:/opt/cray/mpt/7.1.1/gni/bin:/opt/cray/atp/1.7.5/bin:/opt/cray/rca/1.0.0-2.0502.53711.3.127.ari/bin:/opt/cray/alps/5.2.1-2.0502.9041.11.6.ari/sbin:/opt/cray/alps/5.2.1-2.0502.9041.11.6.ari/bin:/opt/cray/dvs/2.5_0.9.0-1.0502.1873.1.145.ari/bin:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/bin:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/bin:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/bin:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/bin:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/bin:/opt/intel/composer_xe_2015.1.133/bin/intel64:/opt/cray/craype/2.2.1/bin:/opt/cray/switch/1.0-1.0502.54233.2.96.ari/bin:/opt/cray/eslogin/eswrap/1.1.0-1.020200.1130.0/bin:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/modules/3.2.10.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/opt/cray/bin
  PBS_O_QUEUE=debug
  PBS_O_SERVER=edique02
  PBS_O_SHELL=/bin/bash
  PBS_O_SUBMIT_FILTER=/usr/syscom/nsg/sbin/submit_filter
  PBS_O_WORKDIR=/global/project/projectdirs/m1881/hpx/edison/nbody_demo-release
  PBS_QUEUE=debug
  PBS_TASKNUM=1
  PBS_VERSION=TORQUE-5.0.1
  PBS_VNODENUM=0
  PBS_WALLTIME=1800
  PELOCAL_PRGENV=true
  PE_CRAY_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/CRAY/83/lib/pkgconfig:/opt/cray/hdf5-parallel/1.8.13/CRAY/83/lib/pkgconfig:/opt/cray/netcdf/4.3.2/CRAY/83/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.2/CRAY/83/lib/pkgconfig:/opt/cray/parallel-netcdf/1.5.0/CRAY/83/lib/pkgconfig:/opt/cray/hdf5/1.8.13/CRAY/83/lib/pkgconfig
  PE_CXX_PKGCONFIG_LIBS=mpichcxx
  PE_ENV=INTEL
  PE_FFTW_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_FFTW_DEFAULT_TARGET_haswell=haswell
  PE_FFTW_DEFAULT_TARGET_interlagos=interlagos
  PE_FFTW_DEFAULT_TARGET_sandybridge=sandybridge
  PE_FFTW_DEFAULT_TARGET_x86_64=x86_64
  PE_FFTW_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/fftw/3.3.4.1/@PE_FFTW_DEFAULT_TARGET@/lib/pkgconfig
  PE_FORTRAN_PKGCONFIG_LIBS=mpichf90
  PE_GA_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_GA_DEFAULT_GENCOMPS_GNU=49 48
  PE_GA_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/@PRGENV@/@PE_GA_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_GA_DEFAULT_VOLATILE_PRGENV=GNU
  PE_HDF5_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_HDF5_DEFAULT_GENCOMPS_GNU=49 48
  PE_HDF5_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5/1.8.13/@PRGENV@/@PE_HDF5_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_HDF5_DEFAULT_VOLATILE_PRGENV=GNU
  PE_HDF5_PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_HDF5_PARALLEL_DEFAULT_GENCOMPS_GNU=49 48
  PE_HDF5_PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5-parallel/1.8.13/@PRGENV@/@PE_HDF5_PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU
  PE_INTEL_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/ga/5.3.0.1/INTEL/140/lib/pkgconfig:/opt/cray/hdf5-parallel/1.8.13/INTEL/140/lib/pkgconfig:/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib/pkgconfig:/opt/cray/netcdf/4.3.2/INTEL/140/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib/pkgconfig:/opt/cray/parallel-netcdf/1.5.0/INTEL/140/lib/pkgconfig:/opt/cray/hdf5/1.8.13/INTEL/140/lib/pkgconfig
  PE_INTEL_FIXED_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-intel/140/lib/pkgconfig
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_interlagos=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_LIBSCI_DEFAULT_OMP_REQUIRES_openmp=_mp
  PE_LIBSCI_DEFAULT_PKGCONFIG_VARIABLES=PE_LIBSCI_DEFAULT_OMP_REQUIRES_@openmp@
  PE_LIBSCI_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH
  PE_LIBSCI_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/13.0.1/@PRGENV@/@PE_LIBSCI_DEFAULT_GENCOMPS@/@PE_LIBSCI_DEFAULT_TARGET@/lib/pkgconfig
  PE_LIBSCI_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_LIBSCI_GENCOMPS_CRAY_haswell=83
  PE_LIBSCI_GENCOMPS_CRAY_interlagos=83
  PE_LIBSCI_GENCOMPS_CRAY_sandybridge=83
  PE_LIBSCI_GENCOMPS_CRAY_x86_64=83
  PE_LIBSCI_GENCOMPS_GNU_haswell=49 48
  PE_LIBSCI_GENCOMPS_GNU_interlagos=49 48
  PE_LIBSCI_GENCOMPS_GNU_sandybridge=49 48
  PE_LIBSCI_GENCOMPS_GNU_x86_64=49 48
  PE_LIBSCI_GENCOMPS_INTEL_haswell=140
  PE_LIBSCI_GENCOMPS_INTEL_interlagos=140
  PE_LIBSCI_GENCOMPS_INTEL_sandybridge=140
  PE_LIBSCI_GENCOMPS_INTEL_x86_64=140
  PE_LIBSCI_MODULE_NAME=cray-libsci/13.0.1
  PE_LIBSCI_OMP_REQUIRES_openmp=_mp
  PE_LIBSCI_PKGCONFIG_LIBS=libsci_mpi:libsci
  PE_LIBSCI_PKGCONFIG_VARIABLES=PE_LIBSCI_OMP_REQUIRES_@openmp@
  PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH
  PE_LIBSCI_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/13.0.1/@PRGENV@/@PE_LIBSCI_GENCOMPS@/@PE_LIBSCI_TARGET@/lib/pkgconfig
  PE_LIBSCI_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_MPICH_DEFAULT_DIR_CRAY_DEFAULT64=64
  PE_MPICH_DEFAULT_FIXED_PRGENV=INTEL
  PE_MPICH_DEFAULT_GENCOMPS_CRAY=83
  PE_MPICH_DEFAULT_GENCOMPS_GNU=49 48
  PE_MPICH_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-@PRGENV@@PE_MPICH_DEFAULT_DIR_DEFAULT64@/@PE_MPICH_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_MPICH_DEFAULT_VOLATILE_PRGENV=CRAY GNU
  PE_MPICH_DIR_CRAY_DEFAULT64=64
  PE_MPICH_FIXED_PRGENV=INTEL
  PE_MPICH_GENCOMPS_CRAY=83
  PE_MPICH_GENCOMPS_GNU=49 48
  PE_MPICH_MODULE_NAME=cray-mpich
  PE_MPICH_MULTITHREADED_LIBS_multithreaded=_mt
  PE_MPICH_NV_LIBS_nvidia20=-lcudart
  PE_MPICH_NV_LIBS_nvidia35=-lcudart
  PE_MPICH_PKGCONFIG_VARIABLES=PE_MPICH_NV_LIBS_@accelerator@:PE_MPICH_MULTITHREADED_LIBS_@multithreaded@
  PE_MPICH_TARGET_VAR_nvidia20=-lcudart
  PE_MPICH_TARGET_VAR_nvidia35=-lcudart
  PE_MPICH_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/mpich2-@PRGENV@@PE_MPICH_DIR_DEFAULT64@/@PE_MPICH_GENCOMPS@/lib/pkgconfig
  PE_MPICH_VOLATILE_PRGENV=CRAY GNU
  PE_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_NETCDF_DEFAULT_GENCOMPS_GNU=49 48
  PE_NETCDF_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5
  PE_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf/4.3.2/@PRGENV@/@PE_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU
  PE_NETCDF_HDF5PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS_GNU=49 48
  PE_NETCDF_HDF5PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5_PARALLEL PE_MPICH
  PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf-hdf5parallel/4.3.2/@PRGENV@/@PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU
  PE_PAPI_DEFAULT_ACCEL_LIBS_nvidia35=,-lcupti,-lcudart,-lcuda
  PE_PAPI_DEFAULT_PKGCONFIG_VARIABLES=PE_PAPI_ACCEL_LIBS_@accelerator@
  PE_PAPI_DEFAULT_TARGET_VAR_nvidia35=,-lcupti,-lcudart,-lcuda
  PE_PARALLEL_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL
  PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS_GNU=49 48
  PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/parallel-netcdf/1.5.0/@PRGENV@/@PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig
  PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_PETSC_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_PETSC_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_PETSC_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_PETSC_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_PETSC_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI:PE_TPSL
  PE_PETSC_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/petsc/3.5.2.1/complex/@PRGENV@/@PE_PETSC_DEFAULT_GENCOMPS@/@PE_PETSC_DEFAULT_TARGET@/lib/pkgconfig
  PE_PETSC_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_PKGCONFIG_DEFAULT_PRODUCTS=PE_LIBSCI:PE_GA:PE_HDF5_PARALLEL:PE_TPSL:PE_MPICH:PE_NETCDF:PE_PETSC:PE_TRILINOS:PE_NETCDF_HDF5PARALLEL:PE_FFTW:PE_PARALLEL_NETCDF:PE_HDF5
  PE_PKGCONFIG_LIBS=darshan-runtime:mpich:sma:AtpSigHandler:libsci_mpi:libsci
  PE_PKGCONFIG_PRODUCTS=PE_MPICH:PE_SMA:PE_LIBSCI
  PE_PKGCONFIG_PRODUCTS_DEFAULT=PE_PAPI
  PE_PRODUCT_LIST=CRAYPE_IVYBRIDGE:CRAY_PMI:CRAY_LIBSCI:CRAYPE:INTEL:CRAY_XPMEM:CRAY_DMAPP:CRAY_UGNI:CRAY_UDREG:CRAY_ALPS:CRAY_LLM
  PE_SMA_DIR_CRAY_DEFAULT64=64
  PE_SMA_DIR_PGI_DEFAULT64=64
  PE_SMA_MODULE_NAME=cray-shmem
  PE_SMA_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/7.1.1/gni/sma@PE_SMA_DIR_DEFAULT64@/lib64/pkgconfig
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_haswell=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_interlagos=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_sandybridge=83
  PE_TPSL_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_TPSL_DEFAULT_GENCOMPS_GNU_haswell=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_interlagos=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_sandybridge=49 48
  PE_TPSL_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_haswell=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_interlagos=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_sandybridge=140
  PE_TPSL_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_TPSL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI
  PE_TPSL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/tpsl/1.4.3/@PRGENV@/@PE_TPSL_DEFAULT_GENCOMPS@/@PE_TPSL_DEFAULT_TARGET@/lib/pkgconfig
  PE_TPSL_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PE_TRILINOS_DEFAULT_GENCOMPS_CRAY_x86_64=83
  PE_TRILINOS_DEFAULT_GENCOMPS_GNU_x86_64=49 48
  PE_TRILINOS_DEFAULT_GENCOMPS_INTEL_x86_64=140
  PE_TRILINOS_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_LIBSCI:PE_TPSL
  PE_TRILINOS_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/trilinos/11.12.1.0/@PRGENV@/@PE_TRILINOS_DEFAULT_GENCOMPS@/@PE_TRILINOS_DEFAULT_TARGET@/lib/pkgconfig
  PE_TRILINOS_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL
  PKGCONFIG_ENABLED=1
  PKG_CONFIG_PATH=/usr/common/usg/darshan/2.3.0/lib/pkgconfig:/opt/cray/pmi/5.0.6-1.0000.10439.140.2.ari/lib64/pkgconfig:/opt/cray/craype/2.2.1/pkg-config:/opt/cray/iobuf/2.0.5/lib/pkgconfig:/opt/cray/xpmem/0.1-2.0502.55507.3.2.ari/lib64/pkgconfig:/opt/cray/dmapp/7.0.1-1.0502.9501.5.219.ari/lib64/pkgconfig:/opt/cray/gni-headers/3.0-1.0502.9684.5.2.ari/lib64/pkgconfig:/opt/cray/ugni/5.0-1.0502.9685.4.24.ari/lib64/pkgconfig:/opt/cray/udreg/2.3.2-1.0502.9275.1.12.ari/lib64/pkgconfig:/opt/cray/alps/5.2.1-2.0502.9072.13.1.ari/lib64/pkgconfig:/opt/cray/atp/1.7.5/lib/pkgconfig
  PKG_CONFIG_PATH_DEFAULT=/opt/cray/papi/5.3.2.1/lib64/pkgconfig
  PMI_FORK_RANK=1
  PMI_GNI_COOKIE=1790312448:1790377984
  PMI_GNI_DEV_ID=0
  PMI_GNI_LOC_ADDR=7832:7832
  PMI_GNI_PTAG=196:197
  PRGENVMODULES=/opt/modules/3.2.10.2/init/.prgenvmodules:PrgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi
  PROFILEREAD=true
  PWD=/global/homes/s/sithhell/project/hpx/edison/nbody_demo-release
  PYTHONPATH=/usr/common/usg/altd/2.0/lib64/python2.6/site-packages:/opt/cray/sdb/1.0-1.0502.55976.5.27.ari/lib64/py
  PYTHONSTARTUP=/etc/pythonstart
  QTDIR=/usr/lib/qt3
  QT_SYSTEM_DIR=/usr/share/desktop-data
  RCLOCAL_BASEOPTS=true
  RCLOCAL_PRGENV=true
  SCRATCH=/scratch2/scratchdirs/sithhell
  SDK_HOME=/usr/lib64/jvm/java
  SHELL=/bin/bash
  SHLVL=2
  SHMEM_ABORT_ON_ERROR=1
  TARGETMODULES=/opt/modules/3.2.10.2/init/.targetmodules:craype-abudhabi:craype-abudhabi-cu:craype-accel-host:craype-accel-nvidia20:craype-accel-nvidia30:craype-accel-nvidia35:craype-barcelona:craype-broadwell:craype-haswell:craype-hugepages128K:craype-hugepages128M:craype-hugepages16M:craype-hugepages256M:craype-hugepages2M:craype-hugepages32M:craype-hugepages4M:craype-hugepages512K:craype-hugepages512M:craype-hugepages64M:craype-hugepages8M:craype-intel-knc:craype-interlagos:craype-interlagos-cu:craype-istanbul:craype-ivybridge:craype-mc12:craype-mc8:craype-mic-knl:craype-network-aries:craype-network-gemini:craype-network-infiniband:craype-network-none:craype-network-seastar:craype-sandybridge:craype-shanghai:craype-target-compute_node:craype-target-local_host:craype-target-native:craype-xeon:xtpe-barcelona:xtpe-interlagos:xtpe-interlagos-cu:xtpe-istanbul:xtpe-mc12:xtpe-mc8:xtpe-network-gemini:xtpe-network-seastar:xtpe-shanghai:xtpe-target-native:xtpe-xeon:/etc/opt/cray/modules/site_targetmodules
  TERM=xterm
  TMPDIR=/scratch2/scratchdirs/sithhell
  TOOLMODULES=/opt/modules/3.2.10.2/init/.toolmodules:apprentice:apprentice2:atp:chapel:cray-lgdb:craypat:craypkg-gen:cray-snplauncher:ddt:gdb:iobuf:papi:perftools:perftools-lite:stat:totalview:xt-craypat:xt-lgdb:xt-papi:xt-totalview:/etc/opt/cray/modules/site_toolmodules
  USER=sithhell
  USERMODULES=/opt/modules/3.2.10.2/init/.usermodules:acml:alps:apprentice:apprentice2:atp:blcr:cce:chapel:cray-ccdb:cray-fftw:cray-ga:cray-hdf5:cray-hdf5-parallel:cray-lgdb:cray-libsci:cray-libsci_acc:cray-mpich:cray-mpich2:cray-mpich-compat:cray-netcdf:cray-netcdf-hdf5parallel:cray-parallel-netcdf:craypat:craype:cray-petsc:cray-petsc-complex:craypkg-gen:cray-shmem:cray-snplauncher:cray-tpsl:cray-trilinos:cudatoolkit:ddt:fftw:ga:gcc:hdf5:hdf5-parallel:intel:iobuf:java:lgdb:libfast:libsci_acc:mpich1:netcdf:netcdf-hdf5parallel:netcdf-nofsync:netcdf-nofsync-hdf5parallel:ntk:onesided:papi:parallel-netcdf:pathscale:perftools:perftools-lite:petsc:petsc-complex:pgi:pmi:PrgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi:stat:totalview:tpsl:trilinos:xt-asyncpe:xt-craypat:xt-lgdb:xt-libsci:xt-mpich2:xt-mpt:xt-papi:xt-shmem:xt-totalview:/etc/opt/cray/modules/site_usermodules
  WINDOWMANAGER=
  XCURSOR_THEME=crystalwhite
  XDG_CONFIG_DIRS=/etc/xdg
  XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share
  XKEYSYMDB=/usr/share/X11/XKeysymDB
  XNLSPATH=/usr/share/X11/nls
  XTOS_VERSION=5.2.40
  XTPE_NETWORK_TARGET=aries
  _=/usr/common/usg/altd/2.0/bin/aprun
  _DMAPPI_RESERVED_MDDS=1
  _LMFILES_=/opt/modulefiles/modules/3.2.10.2:/usr/syscom/nsg/modulefiles/nsg/1.2.0:/opt/modulefiles/hss-llm/7.2.0:/opt/cray/ari/modulefiles/nodestat/2.2-1.0502.53712.3.109.ari:/opt/cray/ari/modulefiles/sdb/1.0-1.0502.55976.5.27.ari:/opt/cray/ari/modulefiles/alps/5.2.1-2.0502.9072.13.1.ari:/opt/cray/modulefiles/lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51:/opt/cray/ari/modulefiles/udreg/2.3.2-1.0502.9275.1.12.ari:/opt/cray/ari/modulefiles/ugni/5.0-1.0502.9685.4.24.ari:/opt/cray/ari/modulefiles/gni-headers/3.0-1.0502.9684.5.2.ari:/opt/cray/ari/modulefiles/dmapp/7.0.1-1.0502.9501.5.219.ari:/opt/cray/ari/modulefiles/xpmem/0.1-2.0502.55507.3.2.ari:/opt/modulefiles/Base-opts/1.0.2-1.0502.53325.1.2.ari:/opt/cray/craype/default/modulefiles/craype-network-aries:/opt/cray/modulefiles/craype/2.2.1:/opt/modulefiles/intel/15.0.1.133:/opt/cray/modulefiles/cray-libsci/13.0.1:/opt/cray/ari/modulefiles/pmi/5.0.6-1.0000.10439.140.2.ari:/opt/cray/modulefiles/atp/1.7.5:/opt/cray/modulefiles/PrgEnv-intel/5.2.40:/opt/cray/craype/default/modulefiles/craype-ivybridge:/opt/cray/modulefiles/cray-shmem/7.1.1:/opt/cray/modulefiles/cray-mpich/7.1.1:/opt/modulefiles/torque/5.0.1:/opt/modulefiles/moab/8.0.1-2014110616-5c7a394-sles11:/usr/common/usg/Modules/modulefiles/altd/2.0:/usr/common/usg/Modules/modulefiles/darshan/2.3.0:/usr/common/usg/Modules/modulefiles/usg-default-modules/1.1:/opt/modulefiles/gcc/4.9.2:/usr/common/usg/Modules/modulefiles/boost/1.54
{locality-id}: 1
{hostname}: [ (mpi:1) ]
{process-id}: 37597
{function}: create_counter_local
{file}: /project/projectdirs/m1881/hpx/source/src/performance_counters/counters.cpp
{line}: 680
{os-thread}: 14, worker-thread#17
{thread-id}: 00002aaac1c217a0
{thread-description}: <unknown>
{state}: startup
{auxinfo}: 
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_PARCELPORT_TCP=OFF
  HPX_PARCELPORT_MPI=ON (MPICH V3.1.2, MPI V3.0)
  HPX_PARCELPORT_IPC=OFF
  HPX_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_RUN_MAIN_EVERYWHERE=OFF
  HPX_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX (configured)=/global/homes/s/sithhell/project/hpx/edison/hpx-release
  HPX_PREFIX=/global/homes/s/sithhell/project/hpx/edison/hpx-release
{version}: V0.9.10-rc1 (AGAS: V3.0), Git: 17b2f0c4d88a939359ee6d521694fd6ba00d72c4
{boost}: V1.56.0
{build-type}: release
{date}: Mar 16 2015 12:12:46
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1500
{stdlib}: GNU libstdc++ version 20141030
{what}: no create function for performance counter found: /agas{locality#1/total}/count/resolve_id (counter type is not defined: HPX(bad_parameter)): HPX(bad_parameter):
hkaiser commented 9 years ago

This has been fixed by merging #1417