search

STEllAR-GROUP / hpx

The C++ Standard Library for Parallelism and Concurrency
https://hpx.stellar-group.org
Boost Software License 1.0
2.54k stars 439 forks source link

HPX does not uses distributed localities on Fugaku #6299

Closed diehlpk closed 1 year ago

diehlpk commented 1 year ago

Expected Behavior

HPX should use all available nodes provided by the job script.

Actual Behavior

HPX ignores the additional nodes and only runs on a single node and not using the other nodes.

This reflects in the computation time which is around the same on a single node, two nodes, and four nodes.

For distributed runs, Octo-Tiger would write as many output files as nodes. However, only one file is written for distributed runs.

Specifications

... Please describe your environment

diehlpk commented 1 year ago

Now, I get this error

[WARN] PLE 0610 plexec The process terminated with the signal.(rank=0)(nid=0x22570004)(sig=11)

diehlpk commented 1 year ago

That is the env 

[u10393@i25-1200c runs_sc23]$ env
PJM_LLIO_LOCALTMP_SIZE=0
LD_LIBRARY_PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/lib64:/vol0004/apps/oss/mpigcc/fjmpi-gcc11/lib:/vol0004/apps/oss/mpigcc/fjmpi-gcc11/lib64:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib64:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/lib:/opt/FJSVxtclanga/tcsds-1.2.37/lib64
LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
SPACK_PYTHON=/usr/bin/python3
INCLUDE=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/include
PJM_JOBNAME=STDIN
PJM_MPI_PROC=2
PJM_RSCUNIT=rscunit_ft01
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
LANG=C
PJM_O_MAIL=/var/spool/mail/u10393
HISTCONTROL=ignoredups
PJM_PK_retention_state_max=1
HOSTNAME=i25-1200c
PJM_ELAPSED_TIME_MODE=fixed
PJM_O_LOGNAME=u10393
C_INCLUDE_PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/include
PJM_LLIO_GFSCACHE_MERGE=/vol0004:/vol0003
PATH_modshare=/usr/sbin:1:/usr/bin:1:/opt/local/mpiexec:1:/opt/FJSVxtclanga/tcsds-1.2.37/bin:1:/usr/local/sbin:1:/usr/share/Modules/bin:1:/usr/local/bin:1
LOADEDMODULES_modshare=lang/tcsds-1.2.37:1
PJM_NORESTART=1
PJM_SUBJOBID=25596321
PJM_PK_eco_state_min=0
PJM_PK_issue_state_min=0
OMPI_CKPTCONFIG=0x00
PJM_PK_retention_state_min=0
PJM_LLIO_GFSCACHE=/vol0004
PJM_STDOUT_PATH=/vol0004/ra010008/data/u10393/OctoTigerBuildChain/STDIN.25596321.out
PJM_NODE_ALLOCATION_MODE=noncont
PJM_NODE_QUOTA_LIMIT=18446744073709551615
PJM_O_HOME=/home/u10393
S_COLORS=auto
which_declare=declare -f
PJM_LLIO_AUTO_READAHEAD=on
PJM_PK_eco_state_max=2
LD_LIBRARY_PATH_modshare=/opt/FJSVxtclanga/tcsds-1.2.37/lib64:1
CC=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/bin/gcc
MPIF90=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/bin/mpifort
PJM_LLIO_STRIPE_COUNT=24
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
PJM_PK_throttling_state_min=0
USER=u10393
PJM_NODE=2
PJM_PK_issue_state_max=1
PJM_JOBID=25596321
FJSVXTCLANGA=/opt/FJSVxtclanga/tcsds-1.2.37
PJM_O_SHELL=/bin/bash
MPICXX=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/bin/mpicxx
PJM_DPREFIX=#PJM
PJM_STDERR_PATH=/vol0004/ra010008/data/u10393/OctoTigerBuildChain/STDIN.25596321.err
PJM_NODE_MEM_LIMIT=18446744073709551615
PWD=/vol0004/ra010008/data/u10393/runs_sc23
MODULES_LMCONFLICT_modshare=lang/tcsds-1.2.37&lang:1
HOME=/home/u10393
CMAKE_PREFIX_PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn
PJM_ELAPSE_LIMIT=1500
PJM_PK_throttling_state_max=9
PLE_SCRIPT_TYPE=JOB_INTERACTIVE
OPAL_PREFIX=/vol0004/apps/oss/mpigcc/fjmpi-gcc11
PJM_LLIO_ASYNC_CLOSE=off
F77=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/bin/gfortran
PJM_O_WORKDIR=/vol0004/ra010008/data/u10393/OctoTigerBuildChain
PJM_LLIO_CN_CACHED_WRITE_SIZE=67108864
PJM_MAILOPTION=0x0
XOS_MMM_L_PAGING_POLICY=demand:demand:prepage
_LMFILES__modshare=/work/system/modulefiles/lang/tcsds-1.2.37:1
MODULES_LMALTNAME=lang/tcsds-1.2.37&lang/default&lang
PJM_O_LANG=C
PJM_LLIO_ACTIVATE=/vol0001:/vol0002:/vol0003:/vol0004:/vol0005:/vol0006
PJM_LOCALTMP=/local
FJPNCKPT_CONFIG=0x00
PJM_RSCGRP=int
PJM_PK_issue_state=0
PJM_O_PATH=/opt/FJSVxtclanga/tcsds-1.2.37/bin:/opt/local/openssh/bin:/usr/share/Modules/bin:/opt/local/openssh/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/u10393/.local/bin:/home/u10393/bin
TMPDIR=/home/u10393
LIBRARY_PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/lib64:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib64:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/lib
LOADEDMODULES=lang/tcsds-1.2.37
PJM_NET_ROUTE=static
FC=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/bin/gfortran
PJM_PK_freq_min=2000
PJM_O_HOST=fn01sv06
MPICC=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/bin/mpicc
PJM_LLIO_CN_READ_CACHE=on
MODULES_LMCONFLICT=lang/tcsds-1.2.37&lang
MAIL=/var/spool/mail/u10393
CXX=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/bin/g++
PJM_PK_eco_state=0
SPACK_ROOT=/vol0004/apps/oss/spack-v0.17.0
TERM=xterm-256color
PJM_CUSTOM_RESOURCES=estimated-power=218,exepjrsh=0,nnum_cret=0,test_rsc01=0,eco_state=0,eco_state_max=2,eco_state_min=0,ex_pipe_state=0,ex_pipe_state_max=1,ex_pipe_state_min=0,freq=2000,freq_max=2200,freq_min=2000,issue_state=0,issue_state_max=1,issue_state_min=0,jobenv=linux2,knob_io=SYSTEM,retention_state=1,retention_state_acores=0,retention_state_max=1,retention_state_min=0,throttling_state=0,throttling_state_max=9,throttling_state_min=0
PJM_SHAREDTMP=/share
PJM_PK_freq_max=2200
SPACK_LOADED_HASHES=cvur4ouyjez2esed6crxy7cwjveqgmcc:nphnrhliclp47njgp2orlwyriezsksdl:sza3atjm2pyuizapqulqortymfjsmckd:v7ajonl2u7ou5faslslrh3jm26j3ho2v:4g4hbl27xwkpapburwgimzegm74pmdme:hmyqduusgd4tpaawe2ckb7e6w62vvgbn:etgwkgytyok3774azi7ignhmspvqtcee
PJM_LLIO_SHAREDTMP_SIZE=0
MODULES_LMALTNAME_modshare=lang/tcsds-1.2.37&lang/default&lang:1
FJSVXTCMPI=/opt/FJSVxtclanga/tcsds-1.2.37
PJM_NODE_ALLOCATION_IO_MODE=no-io-exclusive
SHLVL=1
MPIF77=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/bin/mpifort
ACLOCAL_PATH=/usr/share/aclocal:/usr/share/aclocal
PJM_NODE_X=2
PJM_JOBDIR=/vol0004/ra010008/data/u10393/OctoTigerBuildChain
MANPATH=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/share/man:/usr/share/man:/usr/share/man::
PJM_PK_ex_pipe_state=0
PJM_LLIO_SIO_READ_CACHE=on
PJM_PK_freq=2000
SPACK_LD_LIBRARY_PATH=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib64:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/lib:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/lib:/opt/FJSVxtclanga/tcsds-1.2.37/lib64
MODULEPATH=/vol0004/apps/oss/spack-v0.17.0/share/spack/modules/linux-rhel8-a64fx:/work/system/modulefiles
PJM_NODE_CPUTIME_LIMIT=18446744073709551615
PJM_PK_ex_pipe_state_min=0
LOGNAME=u10393
PJM_ENVIRONMENT=INTERACT
CPLUS_INCLUDE_PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpc-1.1.0-4g4hbl27xwkpapburwgimzegm74pmdme/include:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/include
MODULEPATH_modshare=/work/system/modulefiles:2
PJM_PK_knob_io=SYSTEM
PJM_LLIO_PERF=off
PJM_SHELL=/bin/bash
PJM_LLIO_CN_CACHE_SIZE=134217728
PATH=/vol0004/apps/oss/mpigcc/fjmpi-gcc11/bin:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/gcc-11.2.0-nphnrhliclp47njgp2orlwyriezsksdl/bin:/vol0004/apps/oss/spack-v0.17.0/bin:/opt/local/mpiexec:/opt/FJSVxtclanga/tcsds-1.2.37/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/u10393/.local/bin:/home/u10393/bin
_LMFILES_=/work/system/modulefiles/lang/tcsds-1.2.37
PJM_LLIO_UNCOMPLETED_FILEINFO_PATH=/vol0004/ra010008/data/u10393/OctoTigerBuildChain/STDIN.25596321.err
PJM_LLIO_STRIPE_SIZE=2097152
MODULESHOME=/usr/share/Modules
PKG_CONFIG_PATH=/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/zstd-1.5.0-sza3atjm2pyuizapqulqortymfjsmckd/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/vol0004/apps/oss/spack-v0.17.0/opt/spack/linux-rhel8-a64fx/gcc-8.4.1/mpfr-4.1.0-hmyqduusgd4tpaawe2ckb7e6w62vvgbn/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig
HISTSIZE=1000
PJM_LLIO_IGNORE=/.vol0001:/.vol0002:/.vol0005:/.vol0006
PJM_PK_retention_state=1
PJM_PK_throttling_state=0
PJM_PROC_BY_NODE=1
PJM_PK_ex_pipe_state_max=1
LESSOPEN=||/usr/bin/lesspipe.sh %s
PJM_MPI_SHAPE_X=2
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_module%%=() {  _module_raw "$@" 2>&1
}
BASH_FUNC_spack%%=() {  : this is a shell function from: /vol0004/apps/oss/spack-v0.17.0/share/spack/setup-env.sh;
 : the real spack script is here: /vol0004/apps/oss/spack-v0.17.0/bin/spack;
 _spack_shell_wrapper "$@";
 return $?
}
BASH_FUNC__module_raw%%=() {  unset _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 unset _mlre _mlIFS;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 eval `eval ${_mlre} /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
 else
 eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
 fi;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 unset _mlre _mlv _mlrv _mlIFS;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlshdbg;
 return $_mlstatus
}
BASH_FUNC__spack_shell_wrapper%%=() {  for var in LD_LIBRARY_PATH DYLD_LIBRARY_PATH DYLD_FALLBACK_LIBRARY_PATH;
 do
 eval "if [ -n \"\${${var}-}\" ]; then export SPACK_$var=\${${var}}; fi";
 done;
 if [ -n "${ZSH_VERSION:-}" ]; then
 emulate -L sh;
 fi;
 _sp_flags="";
 while [ ! -z ${1+x} ] && [ "${1#-}" != "${1}" ]; do
 _sp_flags="$_sp_flags $1";
 shift;
 done;
 if [ -n "$_sp_flags" ] && [ "${_sp_flags#*h}" != "${_sp_flags}" ] || [ "${_sp_flags#*V}" != "${_sp_flags}" ]; then
 command spack $_sp_flags "$@";
 return;
 fi;
 _sp_subcommand="";
 if [ ! -z ${1+x} ]; then
 _sp_subcommand="$1";
 shift;
 fi;
 case $_sp_subcommand in 
 "cd")
 _sp_arg="";
 if [ -n "$1" ]; then
 _sp_arg="$1";
 shift;
 fi;
 if [ "$_sp_arg" = "-h" ] || [ "$_sp_arg" = "--help" ]; then
 command spack cd -h;
 else
 LOC="$(spack location $_sp_arg "$@")";
 if [ -d "$LOC" ]; then
 cd "$LOC";
 else
 return 1;
 fi;
 fi;
 return
 ;;
 "env")
 _sp_arg="";
 if [ -n "$1" ]; then
 _sp_arg="$1";
 shift;
 fi;
 if [ "$_sp_arg" = "-h" ] || [ "$_sp_arg" = "--help" ]; then
 command spack env -h;
 else
 case $_sp_arg in 
 activate)
 _a=" $@";
 if [ -z ${1+x} ] || [ "${_a#* --sh}" != "$_a" ] || [ "${_a#* --csh}" != "$_a" ] || [ "${_a#* -h}" != "$_a" ] || [ "${_a#* --help}" != "$_a" ]; then
 command spack env activate "$@";
 else
 stdout="$(command spack $_sp_flags env activate --sh "$@")" || return;
 eval "$stdout";
 fi
 ;;
 deactivate)
 _a=" $@";
 if [ "${_a#* --sh}" != "$_a" ] || [ "${_a#* --csh}" != "$_a" ]; then
 command spack env deactivate "$@";
 else
 if [ -n "$*" ]; then
 command spack env deactivate -h;
 else
 stdout="$(command spack $_sp_flags env deactivate --sh)" || return;
 eval "$stdout";
 fi;
 fi
 ;;
 *)
 command spack env $_sp_arg "$@"
 ;;
 esac;
 fi;
 return
 ;;
 "load" | "unload")
 _a=" $@";
 if [ "${_a#* --sh}" != "$_a" ] || [ "${_a#* --csh}" != "$_a" ] || [ "${_a#* -h}" != "$_a" ] || [ "${_a#* --list}" != "$_a" ] || [ "${_a#* --help}" != "$_a" ]; then
 command spack $_sp_flags $_sp_subcommand "$@";
 else
 stdout="$(command spack $_sp_flags $_sp_subcommand --sh "$@")" || return;
 eval "$stdout";
 fi
 ;;
 *)
 command spack $_sp_flags $_sp_subcommand "$@"
 ;;
 esac
}
BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/share/Modules/libexec/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/share/Modules/libexec/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/Modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}
BASH_FUNC_ml%%=() {  module ml "$@"
}
_=/usr/bin/env
OLDPWD=/vol0004/ra010008/data/u10393
hkaiser commented 1 year ago

Could you also give me the environment generated by mpiexec, for both localities, please?

diehlpk commented 1 year ago

@hkaiser It seems running mpirun env results in an empty result.

Even storing the result of mpirun in a variable seems to be empty

$ mpirun env
$ res=$(mpirun env)
$ echo $res

So I can run env without mpirun and will get the env of both nodes.

diehlpk commented 1 year ago

On differens is that PMIX_RANK which we used before is not set in the env, see here

env | grep PMIX_RANK which results in not found.

But we read this environment variable here

https://github.com/STEllAR-GROUP/hpx/blob/c16ae1091ca5ff3cf880cee49bbf265a3f5d7f8a/libs/core/batch_environments/src/pjm_environment.cpp#L47C48-L47C57

hkaiser commented 1 year ago

What alternative variables can we use to detect the rank a process is running as.

diehlpk commented 1 year ago

Using --hpx:ignore-batch-env seems to work. There seems to be something off with the MPI Environment on Fugaku.