search

UCL-RITS / rcps-buildscripts

Scripts to automate package builds on RC Platforms
MIT License
39 stars 26 forks source link

Install Request: Chemshell 23.0.1 with Python (local install) #582

Open NicoleLabrAvila opened 1 month ago

NicoleLabrAvila commented 1 month ago

Application: Chemshell 23.0.1

Link: Installing ChemShell — Py-ChemShell 23.0.0 documentation

Cluster: Myriad (user account)

Description: Chemshell version that uses Python

License: open source

Special versions or variants: Needs to compile it with NAMD 2.14 or superior

Ticket number: IN06730642

This is my script:

###############################################
#!/usr/bin/env bash

##############################################
########### CHANGE YOUR USER HERE ##########
##############################################
user="skgtnl1"
##############################################

cd /shared/ucl/apps/build_scripts/
source includes/source_includes.sh
module purge

work_dir="/home/$user/pychems"
cd ~/
mkdir "$work_dir"
cd $work_dir

##############################################
######### Installing Py-Chemshell #################
##############################################
echo "Building Chemshell"

module purge
require gcc-libs
require compilers/intel/2018/update3
require mpi/intel/2018/update3/intel
require tcl
require cmake/3.13.3
require python/3.8.0

cd $work_dir
package_name="chemsh-py"
package_version="23.0.1"
make_build_env --tmp-root=/home/$user/pychems

wget "https://chemshell.org/wp-content/uploads/2023/12/chemsh-py-23.0.1.tar.gz"

SRC_ARCHIVE=${SRC_ARCHIVE:-$work_dir/chemsh-py-23.0.1.tar.gz}
install_prefix="$work_dir/chems-py-23.0.1"
module_dir="$work_dir/chems-py-23.0.1_module"
mkdir -p "$install_prefix"
cd "$install_prefix"
tar zxvf "$SRC_ARCHIVE"
rm $SRC_ARCHIVE

############### Build #########################
./chemsh-py-23.0.1/setup --mpi -rb --fc mpiifort --cc mpiicc -cpc mpicicpc --charmm-ff --namd /shared/ucl/apps/pkg-store/NAMD_2.14_Linux-x86_64-multicore-CUDA.tar.gz
#####################################################

make_module_v2 \
    -d \
    -r gcc-libs \
    -r compilers/intel/2018/update3 \
    -r require mpi/intel/2018/update3/intel \
    -r require tcl \
    -r cmake/3.13.3 \
    -r python/3.8.0 \
    -e "$install_prefix"/chems-py-23.0.1

chmod -R +rX "$module_dir"

rm -Rf "${build_dir:?error: tried to rm blank}"
rm -rf $work_dir/chems-py-t*

post_build_report
NicoleLabrAvila commented 1 month ago

Build always asks for charm. I tried to compile one of the NAMD with charm codes existent here, in the user's home but it didn't work because charm build was sending errors.

In the Build line in the script I did, the --charmm-ff is an option available to install charm but it doesn't work either.

This is the error I get:

CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error at /lustre/shared/ucl/apps/cmake/3.13.3/gnu-4.9.2/share/cmake-3.13/Modules/FindMPI.cmake:1187 (try_compile):
  Failed to configure test project build system.
Call Stack (most recent call first):
  /lustre/shared/ucl/apps/cmake/3.13.3/gnu-4.9.2/share/cmake-3.13/Modules/FindMPI.cmake:1228 (_MPI_try_staged_settings)
  /lustre/shared/ucl/apps/cmake/3.13.3/gnu-4.9.2/share/cmake-3.13/Modules/FindMPI.cmake:1488 (_MPI_check_lang_works)
  CMakeLists.txt:283 (find_package)

-- Configuring incomplete, errors occurred!
See also "/lustre/home/skgtnl1/pychems/chems-py-23.0.1/chemsh-py-23.0.1/chemsh/CMakeFiles/CMakeOutput.log".
See also "/lustre/home/skgtnl1/pychems/chems-py-23.0.1/chemsh-py-23.0.1/chemsh/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "./chemsh-py-23.0.1/setup", line 1095, in <module>
    main()
  File "./chemsh-py-23.0.1/setup", line 1073, in main
    subprocess.run(getCMakeCmd(args),
  File "/shared/ucl/apps/python/3.8.0/gnu-4.9.2/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'cmake -DCMAKE_Fortran_COMPILER:FILEPATH=mpiifort -DCMAKE_C_COMPILER:FILEPATH=mpiicc -DCHEMSH_ARCH:STRING= -DCMAKE_CXX_COMPILER:FILEPATH=mpicicpc -DHOSTNAME:STRING=login13 -DCHEMSH_MAKE_NJOBS:STRING=4 -DCHEMSH_64_BIT:BOOL=OFF -DCHEMSH_Fortran_FLAGS:STRING="" -DCHEMSH_C_FLAGS:STRING="" -DCHEMSH_LD_FLAGS:STRING="" --no-warn-unused-cli -DCHEMSH_COMPILE_OPT_LEVEL:STRING=2 -DLD_LIBRARY_PATH="-L/opt/intel/lib/intel64" -DCHEMSH_LD_LIBRARY_PATH:PATH="/opt/intel/lib/intel64" -DPLATFORM:STRING=ubuntu -DPYTHONPATH= -DATEN_RECOMPILE:BOOL=OFF -DCASTEP_RECOMPILE:BOOL=OFF -DCHARM++_RECOMPILE:BOOL=OFF -DCP2K_RECOMPILE:BOOL=OFF -DDL_MONTE_RECOMPILE:BOOL=OFF -DDL_POLY_RECOMPILE:BOOL=OFF -DGA_RECOMPILE:BOOL=OFF -DGAMESS-UK_RECOMPILE:BOOL=OFF -DGULP_RECOMPILE:BOOL=OFF -DLSDALTON_RECOMPILE:BOOL=OFF -DNAMD_RECOMPILE:BOOL=OFF -DNWCHEM_RECOMPILE:BOOL=OFF -DRECOMPILE_WITHOUT_DOWNLOAD:BOOL=OFF -DGA= -DGA_RECONFIGURE=ON -DDL_FIELD= -DGULP:PATH= -DGAMESS-UK:PATH= -DNAMD:PATH=/lustre/shared/ucl/apps/pkg-store/NAMD_2.14_Linux-x86_64-multicore-CUDA.tar.gz -DCHARMM_FF=http://mackerell.umaryland.edu/download.php?filename=CHARMM_ff_params_files/toppar_c36_jul21.tgz -DFHIAIMS:PATH= -DMATH_HEADER_MACROS="-U__linux__ -U__USE_MISC -U__NetBSD__ -D__FreeBSD__ -U__PURE_INTEL_C99_HEADERS__" -DBLAS_DIR:PATH=/opt/intel/mkl/lib/intel64 -DLAPACK_DIR:PATH=/opt/intel/mkl/lib/intel64 -DFFTW_DIR:PATH=/opt/intel/mkl/interfaces/fftw3xf -DFFTW_INCLUDE_DIR:PATH=/opt/intel/mkl/include/fftw -DScaLAPACK_DIR:PATH=/opt/intel/mkl/lib/intel64 -DMKL=/opt/intel/mkl/lib/intel64 -DMKL_FLAGS:STRING="" -DMKL_COMPILE_FLAGS:STRING="" -DMPI:BOOL=ON -DMPIEXEC_EXECUTABLE:FILEPATH=mpirun --no-warn-unused-cli -Wno-dev .' returned non-zero exit status 1.
Creating module file as: /home/skgtnl1/pychems/chems-py-23.0.1_module/chemsh-py/23.0.1/intel-2018
heatherkellyucl commented 1 month ago

Reading NAMD_2.14_Source/charm-6.10.2/README.md:

CHARMARC in our build script could be multicore-linux-x86_64 for Myriad shared memory jobs. Or it could be ucx-linux-x86_64 but is unlikely to be needed (unless Economics segment of Myriad).

mpi-linux-x86_64 should also work but it suggests it would be slower.

The options come after that, and these may or may not be relevant:

* `smp` - Enable direct SMP support.  An `smp` version communicates using
    shared memory within a process but normal message passing across
    processes and nodes. `smp` mode also introduces a dedicated 
    communication thread for every process. Because of locking, `smp` may
    slightly impact non-SMP performance. Try your application to decide if
    enabling `smp` mode improves performance.
* `tcp` - The `netlrts-` version communicates via UDP by default. The `tcp` option
    will use TCP instead. The TCP version of Charm++ is usually slower
    than UDP, but it is more reliable.

You can view the options for the chosen arch (so we'd want to look at multicore-linux for example:

        $ ./build charm++ netlrts-linux help

     will give:

      Supported compilers: clang craycc gcc icc iccstatic msvc pgcc xlc xlc64
      Supported options: bigemulator bigsim clustermatic common cuda flang gfortran ifort local nolb omp ooc papi pedantic perftools persistent pgf90 pxshm scyld smp syncft sysvshm tcp tsan
NicoleLabrAvila commented 3 weeks ago

The following code successfully installs Chemshell with Python and NAMD 2.14 + Charm++ + GPU. The binary files are generated correctly and the only part that does not work well is the module creation, at the end. The module is created but doesn't work well. It seems environment variables are not correctly set. 

#!/usr/bin/env bash

###############################################
# Installing PyChemShell 23.0.1 parallel GPU version 
# Built with Intel compiler without MPI, no needed for Myriad
# Source for NAMD already existent 

##############################################
########### CHANGE YOUR USER HERE ############
##############################################
USER=ucqntbi
##############################################
cd /shared/ucl/apps/build_scripts/
source includes/source_includes.sh

# Setting working directory 
echo "USER: $USER"
work_dir="/home/$USER/pychems"
cd ~/
mkdir "$work_dir"
cd $work_dir

module purge
#require beta-modules
require gcc-libs/7.3.0
#require gcc-libs/10.2.0
require compilers/intel/2019/update5
require cuda/11.3.1/gnu-10.2.0
require mpi/intel/2019/update5/intel
require cmake/3.13.3
require python/3.8.0

##############################################
############## Installing NAMD ###############
##############################################

echo "Buildig NAMD 2.14"
SRC_ARCHIVE=${SRC_ARCHIVE:-/shared/ucl/apps/pkg-store/NAMD_2.14_Source.tar.gz}
package_name="namd"
package_version="2.14"

CHARMVER=${CHARMVER:-6.10.2}
CHARMARC=${CHARMARC:-"multicore-linux-x86_64"}

make_build_env --tmp-root=/home/$USER/pychems

install_prefix="$work_dir/NAMD_2.14" 
module_dir="$work_dir/NAMD_2.14_module"
mkdir -p "$install_prefix"
cd "$install_prefix"
tar zxvf "$SRC_ARCHIVE"
echo "tar Done... Building Charm++"

ln -fsT `pwd`/NAMD_2.14_Source ${install_prefix}/NAMD_2.14_Source

cd ${install_prefix}/NAMD_2.14_Source

# Fix for charm not being able to find f90
mkdir uclbuildsupport
ln -s `which $FC` uclbuildsupport/f90
ln -s `which $FC` uclbuildsupport/f77
ln -s `which $FC` uclbuildsupport/g77

export PATH=`pwd`/uclbuildsupport:$PATH

# Needs libmpi.so to be directly in the library path
export LD_LIBRARY_PATH=$MPI_HOME/intel64/lib/release_mt:$LD_LIBRARY_PATH

echo "Installing CHARM++"
tar xvf charm-${CHARMVER}.tar
export MPICXX=mpicxx

cd charm-${CHARMVER}
./build charm++ ${CHARMARC} ifort iccstatic --no-build-shared --with-production -xHost -DCMK_OPTIMIZE

echo "End build ..... entering to patch"
cd ..
cd arch
cp /shared/ucl/apps/build_scripts/namd-patches/*.patch .
patch < Linux-x86_64.tcl.patch
patch < Linux-x86_64.fftw.patch
cp /shared/ucl/apps/build_scripts/namd-patches/Linux-x86_64.fftw3 .

# AMD EPYC: avx and avx2 available
sed -i.bak 's|-axAVX|-march=core-avx2 -qopt-zmm-usage=high|g' Linux-x86_64-icc.arch

echo "Doing config"
echo "${CHARMARC}-ifort-iccstatic"
cd ..
./config Linux-x86_64-icc --with-fftw3 --with-cuda --charm-arch ${CHARMARC}-ifort-iccstatic 

echo "CHARM++ make"
cd Linux-x86_64-icc
make
echo "CHARM++ end make"
cd ${build_dir}/NAMD_2.14_Source/Linux-x86_64-icc
mkdir -p ${install_prefix}/NAMD_2.14_Source/bin
for a in charmrun flipbinpdb flipdcd namd2 psfgen sortreplicas
do
  echo "Installing $a"
  cp $a ${install_prefix}/NAMD_2.14_Source/bin
done

rm -rf $work_dir/namd*

##############################################
######### Installing Py-Chemshell ############
##############################################
echo "Building Chemshell"

cd $work_dir
package_name="chemsh-py"
package_version="23.0.1"
make_build_env --tmp-root=/home/$USER/pychems

install_prefix="$work_dir/chemsh-py-23.0.1"
module_dir="$work_dir/chemsh-py-23.0.1_module"
mkdir $install_prefix
cd $install_prefix
wget "https://chemshell.org/wp-content/uploads/2023/12/chemsh-py-23.0.1.tar.gz"
tar zxvf $work_dir/chemsh-py-23.0.1/chemsh-py-23.0.1.tar.gz

echo "Installing pyscf ..."
pip3 install pyscf

echo "Building chemshell ..."
echo "NAMD: $work_dir/NAMD_2.14/NAMD_2.14_Source"
./chemsh-py-23.0.1/setup --mpi -rb --fc mpiifort --cc mpiicc -cpc mpiicpc --namd $work_dir/NAMD_2.14/NAMD_2.14_Source
echo "End building"

make_module_v2 \
    -d \
    -r gcc-libs/7.3.0 \
    -r compilers/intel/2019/update5 \
    -r mpi/intel/2019/update5/intel \
    -r cmake/3.13.3 \
    -r python/3.8.0 \
    -r cuda/11.3.1/gnu-10.2.0 \
    -e "$install_prefix"/chemsh-py-23.0.1

chmod -R +rX "$module_dir"

rm -rf $work_dir/chemsh-py-b*
rm -rf $work_dir/chemsh-py-t*
post_build_report
tom-bickley commented 2 weeks ago

Hi Nicole,

Thanks for this! Everything seems to compile correctly without errors.

However I’ve been trying to get NAMD to talk with py-chemshell and haven’t had much luck, perhaps I need to update the link which py-chemshell uses at /home/USERNAME/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/interfaces/namd/namd_path.py ?

I have also tried running the NAMD binary by moving to /home/USERNAME/pychems/NAMD_2.14/NAMD_2.14_Source/bin , and running ./namd2 but it gives the following:

./namd2: /shared/ucl/apps/gcc/4.9.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./namd2)
./namd2: /shared/ucl/apps/gcc/4.9.2/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./namd2)

These errors also occur when I run ./namd2 within /home/USERNAME/pychems/NAMD_2.14/NAMD_2.14_Source/Linux-x86_64-icc folder.

Hopefilly this is a simple fix to update some paths?

One way to check if py-chemshell can talk to NAMD is to navigate to the tests folder at

/home/USERNAME/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/tests

and run:

./test –-sets namd

When I run this is says that the external code (NAMD) is not found.

NicoleLabrAvila commented 1 week ago

The error in the previous install is due to the GPU driver version installed in Myriad. It is too old and not compatible with the version of Chemshell we want to install. I built everything again, without including cuda and no errors but I'm still having issues in the test to check NAMD communication with Chemshell. The error I get is:

ubq_sphere.py -------- FAILED --------- F [mpiexec@node-b00a-008.myriad.ucl.ac.uk] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:731): error waiting for event 2.20

Looking at the log, the error says this:

 >>> chemsh.x ERROR: Failed running ubq_sphere.py

Traceback (most recent call last):
  File "ubq_sphere.py", line 27, in <module>
    md.run(dryrun=False)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/tasks/md.py", line 272, in run
    ierror = self.theory.run(MDOptions=self.options, **kwargs)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/dl_py2f/utils/miscutils.py", line 124, in wrapper
    result = proc(*args, **kwargs)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/interfaces/mm.py", line 147, in run
    ierror = run.run(self, cmdpath, stderr=self.logger, **kwargs)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/dl_py2f/utils/miscutils.py", line 145, in wrapper
    result = proc(*args, **kwargs)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/base/run.py", line 262, in run
    ierror = selectcases[type(cmdpath)](theory, cmdpath, options=options, **kwargs)
  File "/lustre/home/skgtnl1/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/chemsh/base/run.py", line 155, in runExec
    exit(ierror)
  File "/shared/ucl/apps/python/3.8.0/gnu-4.9.2/lib/python3.8/_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: 255

 >>> Serial run failed. Exiting...

I do not have any idea of what is failing.

Here, the script I used to built it without CUDA:

#!/usr/bin/env bash

###############################################
# Installing PyChemShell 23.0.1 parallel GPU version
# Built with Intel compiler without MPI, no needed for Myriad
# Source for NAMD already existent

##############################################
########### CHANGE YOUR USER HERE ############
##############################################
USER=$YOUR_USER
##############################################
cd /shared/ucl/apps/build_scripts/
source includes/source_includes.sh

# Setting working directory
echo "USER: $USER"
work_dir="/home/$USER/pychems"
cd ~/
mkdir "$work_dir"
cd $work_dir

module purge
require beta-modules
require gcc-libs/7.3.0
require compilers/intel/2019/update5
require cuda/11.3.1/gnu-10.2.0
require mpi/intel/2019/update5/intel
require cmake/3.13.3
require python/3.8.0
require fftw/3.3.9/gnu-10.2.0

##############################################
############## Installing NAMD ###############
##############################################

echo "Buildig NAMD 2.14"
SRC_ARCHIVE=${SRC_ARCHIVE:-/shared/ucl/apps/pkg-store/NAMD_2.14_Source.tar.gz}
package_name="namd"
package_version="2.14"
package_variant="ofi-smp-gpu"
NANAME=${NANAME:-NAMD_${package_version}_Source}

CHARMVER=${CHARMVER:-6.10.2}
CHARMARC=${CHARMARC:-"multicore-linux-x86_64"}

make_build_env --tmp-root=/home/$USER/pychems

mkdir -p "$install_prefix"

cd "$build_dir"
tar zxvf "$SRC_ARCHIVE"
ln -fsT `pwd`/${NANAME} ${install_prefix}/${NANAME}

cd ${install_prefix}/${NANAME}

# Fix for charm not being able to find f90
mkdir uclbuildsupport
ln -s `which $FC` uclbuildsupport/f90
ln -s `which $FC` uclbuildsupport/f77
ln -s `which $FC` uclbuildsupport/g77
export PATH=`pwd`/uclbuildsupport:$PATH

# Needs libmpi.so to be directly in the library path
export LD_LIBRARY_PATH=$MPI_HOME/intel64/lib/release_mt:$LD_LIBRARY_PATH

echo "Installing CHARM++"
tar xvf charm-${CHARMVER}.tar
export MPICXX=mpicxx

cd charm-${CHARMVER}
./build charm++ ${CHARMARC} ifort iccstatic --no-build-shared --with-production -xHost -DCMK_OPTIMIZE

cd ..
cd arch
cp /shared/ucl/apps/build_scripts/namd-patches/*.patch .
patch < Linux-x86_64.tcl.patch
patch < Linux-x86_64.fftw.patch
cp /shared/ucl/apps/build_scripts/namd-patches/Linux-x86_64.fftw3 .

# AMD EPYC: avx and avx2 available
sed -i.bak 's|-axAVX|-march=core-avx2 -qopt-zmm-usage=high|g' Linux-x86_64-icc.arch

cd ..
./config Linux-x86_64-icc --with-fftw3 --charm-arch ${CHARMARC}-ifort-iccstatic

echo "Make"
cd Linux-x86_64-icc
make
echo "End make"
NAMD_NAME=${NAMD_NAME:-${build_dir}/${NANAME}}

##############################################
######### Installing Py-Chemshell ############
##############################################
echo "Building Chemshell"

cd $work_dir
package_name="chemsh-py"
package_version="23.0.1"
make_build_env --tmp-root=/home/$USER/pychems

rm -rf $work_dir/chemsh-py-b*
rm -rf $work_dir/chemsh-py-t*

# I use my own directory as the name given automatically is weird
install_prefix="$work_dir/chemsh-py-23.0.1"
mkdir $install_prefix
cd $install_prefix
wget "https://chemshell.org/wp-content/uploads/2023/12/chemsh-py-23.0.1.tar.gz"
tar zxvf $work_dir/chemsh-py-23.0.1/chemsh-py-23.0.1.tar.gz

echo "Installing pyscf ..."
pip3 install pyscf

echo "Building chemshell ..."
./chemsh-py-23.0.1/setup --mpi --fc mpiifort --cc mpiicc --cpc mpiicpc --charmm-ff --namd $NAMD_NAME
echo "End building"

Here the script to test the communication between NAMD and Chemshell:

#!/bin/bash -l

# Batch script to run a test to check Chemshell-NAMD interaction..

# Request ten minutes of wallclock time (format hours:minutes:seconds).
#$ -l h_rt=1:00:00

# Request 1 gigabyte of RAM per process (must be an integer followed by M, G, or T)
#$ -l mem=3G

# Request 1 gigabyte of TMPDIR space per node
# (default is 10 GB - remove if cluster is diskless)
#$ -l tmpfs=10G

# Set the name of the job.
#$ -N Py-Ch-NAMD

# Select the MPI parallel environment.
#$ -pe mpi 2

# Set the working directory to somewhere in your scratch space.
# Replace "<your_UCL_id>" with your UCL user ID :
#$ -wd /home/$YOUR_USER/pychems

# Load/unload modules required
module unload gcc-libs
module unload compilers/intel/2018/update3
module unload mpi/intel/2018/update3/intel

module load gcc-libs/7.3.0
module load compilers/intel/2019/update5
module load mpi/intel/2019/update5/intel
module load python/3.8.0
module load cmake/3.13.3

# Run entire test.
cd /home/$YOUR_USER/pychems/chemsh-py-23.0.1/chemsh-py-23.0.1/tests
./test –-sets namd

I would appreciate if someone can take a look! Any help or comment!

Many thanks in advance!

NicoleLabrAvila commented 1 week ago

The error obtained with the cuda version is this:

main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:1745): error parsing parameters
CUDA initialization error on [node-b00a-006.myriad.ucl.ac.uk](http://node-b00a-006.myriad.ucl.ac.uk/): CUDA driver version is insufficient for CUDA runtime version
FATAL ERROR: CUDA error cudaGetDeviceCount(&deviceCount) in file src/DeviceCUDA.C, function initialize, line 135
 on Pe 0 ([node-b00a-006.myriad.ucl.ac.uk](http://node-b00a-006.myriad.ucl.ac.uk/)): CUDA driver version is insufficient for CUDA runtime version