ximik69
commented
9 months ago Below is startup script
#!/bin/bash
#
# job time, change for what your job requires
#SBATCH -t 00:5:00
#$BATCH -A plgzl3a-cpu
# requesting the number of nodes needed
#SBATCH -N 1
#####SBATCH --exclusive
#SBATCH -p plgrid-testing
#SBATCH --tasks-per-node=4
#SBATCH --cpus-per-task=12
# job name
#SBATCH -J JDFTx_test
#
# filenames stdout and stderr - customise, include %j
#SBATCH -o process_%j.out
#SBATCH -e process_%j.err
# write this script to stdout-file - useful for scripting errors
cat $0
# load the modules required for you program - customise for your program
cd $SLURM_SUBMIT_DIR
. /net/people/plgrid/plgigoro/.bashrc
module load jdftx/1.7.0-foss-2021b-mkl
export OMP_DISPLAY_ENV=TRUE
export OMP_DISPLAY_AFFINITY=TRUE
export OMP_NUM_THREADS=12
export OMP_PROC_BIND=TRUE
dbg=""
runit="mpiexec --report-bindings jdftx $dbg -m -i X.in -o X.out"
$runit
shankar1729
Hello dear Shankar.
Despite some good experience with JDFTx recently I ran into trouble with thread placement. I asked local sysadmins for help but after long time playing with diferent options we did not manage to run program properly. I run JDFTx on cluster with Intel processors, managed by slurm. There are 2 sockets per node 2 numa nodes per socket = 4 numa nodes x 12 cores. JDFTx was compiled by gcc 11.2, with mkl, OpenMPI, gsl, libxc and scalapack.
Symptoms: JDFTx starts proper number of threads, but only a few of cores are loaded, others are working for brief amount of time then idle. Example - starting 4 mpi processes 12 threads each leads to 4 loaded cores # 0, 4, 24, 28 with others idling. I tried nonthreaded mkl, recompiling OpenMPI, using --bind-to none in mpiexec/mpirun arguments but nothing changed the situation.
Short summary - MPI processes x 1 thread each - work Only OpenMP treads w/o MPI - works MPI/OpenMP - thread placement problem.
Below is list of libraries, to which JDFTx is linked.
ldd
which jdftxlinux-vdso.so.1 (0x00007ffd88fed000) libjdftx.so => /net/software/testing/software/jdftx/1.7.0-foss-2021b-mkl/lib/libjdftx.so (0x00001522929c3000) libmpi_cxx.so.40 => /net/software/testing/software/OpenMPI/4.1.1-GCC-11.2.0/lib/libmpi_cxx.so.40 (0x0000152293332000) libmpi.so.40 => /net/software/testing/software/OpenMPI/4.1.1-GCC-11.2.0/lib/libmpi.so.40 (0x0000152293206000) libgsl.so.25 => /net/software/testing/software/GSL/2.7-GCC-11.2.0/lib/libgsl.so.25 (0x000015229253b000) libmkl_scalapack_lp64.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_scalapack_lp64.so.1 (0x0000152291e0e000) libmkl_gf_lp64.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_gf_lp64.so.1 (0x0000152291270000) libmkl_blacs_openmpi_lp64.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_blacs_openmpi_lp64.so.1 (0x00001522931bc000) libmkl_intel_lp64.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_intel_lp64.so.1 (0x00001522906d1000) libmkl_gnu_thread.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_gnu_thread.so.1 (0x000015228eb46000) libmkl_core.so.1 => /net/software/testing/software/imkl/2021.4.0/mkl/2021.4.0/lib/intel64/libmkl_core.so.1 (0x000015228a6d8000) libgomp.so.1 => /net/software/testing/software/GCCcore/11.2.0/lib64/libgomp.so.1 (0x0000152293175000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000015228a4b8000) libxc.so.9 => /net/software/testing/software/libxc/5.1.6-GCC-11.2.0/lib/libxc.so.9 (0x0000152289b94000) libstdc++.so.6 => /net/software/testing/software/GCCcore/11.2.0/lib64/libstdc++.so.6 (0x0000152289968000) libm.so.6 => /lib64/libm.so.6 (0x00001522895e6000) libgcc_s.so.1 => /net/software/testing/software/GCCcore/11.2.0/lib64/libgcc_s.so.1 (0x00001522895cc000) libc.so.6 => /lib64/libc.so.6 (0x0000152289207000) libopen-rte.so.40 => /net/software/testing/software/OpenMPI/4.1.1-GCC-11.2.0/lib/libopen-rte.so.40 (0x000015228914f000) libopen-orted-mpir.so => /net/software/testing/software/OpenMPI/4.1.1-GCC-11.2.0/lib/libopen-orted-mpir.so (0x0000152293168000) libopen-pal.so.40 => /net/software/testing/software/OpenMPI/4.1.1-GCC-11.2.0/lib/libopen-pal.so.40 (0x000015228909f000) librt.so.1 => /lib64/librt.so.1 (0x0000152288e97000) libutil.so.1 => /lib64/libutil.so.1 (0x0000152288c93000) libhwloc.so.15 => /net/software/testing/software/hwloc/2.5.0-GCCcore-11.2.0/lib/libhwloc.so.15 (0x0000152288c38000) libpciaccess.so.0 => /net/software/testing/software/libpciaccess/0.16-GCCcore-11.2.0/lib/libpciaccess.so.0 (0x000015229315c000) libxml2.so.2 => /net/software/testing/software/libxml2/2.9.10-GCCcore-11.2.0/lib/libxml2.so.2 (0x0000152288aca000) libdl.so.2 => /lib64/libdl.so.2 (0x00001522888c6000) libz.so.1 => /net/software/testing/software/zlib/1.2.11-GCCcore-11.2.0/lib/libz.so.1 (0x00001522888ad000) liblzma.so.5 => /net/software/testing/software/XZ/5.2.5-GCCcore-11.2.0/lib/liblzma.so.5 (0x0000152288885000) libevent_core-2.1.so.7 => /net/software/testing/software/libevent/2.1.12-GCCcore-11.2.0/lib/libevent_core-2.1.so.7 (0x000015228884e000) libevent_pthreads-2.1.so.7 => /net/software/testing/software/libevent/2.1.12-GCCcore-11.2.0/lib/libevent_pthreads-2.1.so.7 (0x000015228884a000) /lib64/ld-linux-x86-64.so.2 (0x000015229312c000)