PWDFT

PW-DFT development for NWChemEx

Web location: https://ebylaska.github.io/PWDFT/

CMAKE - Generate a Project Buildsystem

cmake -S Nwpw/ -B build
cd build
make

Alternatively you can build :
mkdir build
cd build
cmake ../Nwpw
make

Standard cmake build commands

cmake [<options>] <path-to-source>
$ mkdir build ; cd build
$ cmake ../src

cmake [<options>] -S <path-to-source> -B <path-to-build>
$ cmake -S src -B build

cmake [<options>] <path-to-existing-build>
$ cd build
$ cmake .

Timings

laptop timings

These timings suggest that parallel FFTs should be implemented using hybrid MPI-OpenMP code, and the large DGEMMs should use GPUs. This is somewhat justified, since the cost of parallel FFTs is mostly due to data movement, i.e. FFTs are memory bound rather than computationlly bound. However, we need to test the competiveness of pipelining FFT data to GPUs, and using Stockholm FFT kernels (https://github.com/ebylaska/PWDFT/tree/master/Miscellaneous/programfft), versus an MPI-only algorithm.

Compiling and Running Instructions

Build instructions on ALCF Aurora/Sunspot

Required Modules

module restore
module load cmake

Getting the code and building instructions

export HTTP_PROXY=http://proxy.alcf.anl.gov:3128
export HTTPS_PROXY=http://proxy.alcf.anl.gov:3128
export http_proxy=http://proxy.alcf.anl.gov:3128
export https_proxy=http://proxy.alcf.anl.gov:3128
git config --global http.proxy http://proxy.alcf.anl.gov:3128
git clone https://github.com/ebylaska/PWDFT.git

cd PWDFT
cmake -H. -Bbuild_sycl -DNWPW_SYCL=On -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_COMPILER=icx -DCMAKE_Fortran_COMPILER=ifx ./Nwpw

Running

qsub -l select=1 -l walltime=30:00 -A catalysis_aesp_CNDA -q lustre_scaling -I

export MPIR_CVAR_ENABLE_GPU=0
export OMP_NUM_THREADS=1
mpiexec -n 12 --ppn 12 --cpu-bind list:0-7:8-15:16-23:24-31:32-39:40-47:52-59:60-67:68-75:76-83:84-91:92-99 --mem-bind list:0:0:0:0:0:0:1:1:1:1:1:1 --env OMP_NUM_THREADS=1 gpu_tile_compact.sh ../../build_sycl/pwdft cco-cu_surf30.nw

Instructions for OLCF Frontier

Instructions for NERSC Perlmutter

Build Instructions on NERSC Cori-Haswell

required modules on Cori-Haswell

module purge
module load cmake/3.20.2
module load PrgEnv-intel
module load craype-haswell
module load openmpi

module purge; module load cmake/3.20.2; module load PrgEnv-intel; module load craype-haswell; module load openmpi

Build Instructions on Cori-Haswell (starting from PWDFT directory)

mkdir build
cd build
cmake -DCMAKE_CXX_COMPILER=CC ../Nwpw/

Running on Cori-Haswell

salloc --nodes 1 --qos interactive --time 01:00:00 --constraint haswell
srun -n <num_mpi_processes> -c <cpus_per_task> a.out
srun -n <num_mpi_processes> -c <cpus_per_task> pwdft

runnin on haswell??

salloc --nodes 1 --qos interactive --time 01:00:00 --constraint haswell
cd PWDFT/QA/C2_steepest_descent
srun -n 24 ../../build/pwdft c2-sd.nw

Instructions for ALCF Polaris

Build instructions on NERSC Cori-CUDA

Required Modules on Cori-CUDA

module unload impi
module load PrgEnv-intel
module load cmake
module load cudatoolkit

Build Instructions on Cori-CUDA (starting from PWDFT directory)

mkdir build_cuda
cd build-cuda
cmake -DNWPW_CUDA=ON ../Nwpw/

Running on Cori-CUDA

module load cgpu
salloc -C gpu -t 60 -c 10 -G 1 -q interactive -A <account>
salloc -C gpu -t 60 -c 10 -G 1 -q interactive -A mp119

Making shared library

To generate a library clean the build directory and then regenerate cmake with

cmake ../Nwpw -DMAKE_LIBRARY=true

on Linux use:

cmake ../Nwpw -DMAKE_LIBRARY=true -DCMAKE_POSITION_INDEPENDENT_CODE=ON

Compile and generate the shared library in the build directory.

make

The shared library, libpwdft.dylib, should be generated and present in the build directory.

prompt% ls
CMakeCache.txt       Makefile             cmake_install.cmake  nwpwlib/
CMakeFiles/          NwpwConfig.h         libpwdft.dylib*      pspw/

Example header to make function calls

#include <string>
#include "mpi.h"

namespace pwdft {
using namespace pwdft;

extern char *util_date();
extern void seconds(double *);
extern int cpsd(MPI_Comm, std::string&);
extern int cpmd(MPI_Comm, std::string&);
extern int pspw_minimizer(MPI_Comm, std::string&);
extern int pspw_geovib(MPI_Comm, std::string&);
}

Example function call

ierr += pwdft::pspw_geovib(MPI_COMM_WORLD,nwinput);

Using shared library on a MAC

mpic++ test.cpp ../build-shared/libpwdft.dylib
setenv DYLD_LIBRARY_PATH /Users/bylaska/Codes/PWDFT/build-shared
a.out

Using shared library on LINUX

mpic++ test.cpp ../build-shared/libpwdft.so
setenv DYLD_LIBRARY_PATH /Users/bylaska/Codes/PWDFT/build-shared
a.out

[2:17 PM] Bagusetty, Abhishek add_library(pwdft SHARED nwpw.cpp)

[2:18 PM] Bagusetty, Abhishek CMakeLists.txt (right after this line add_executable(pwdft nwpw.cpp))