Simulation on HPC

[Weiming-Hu]

XSEDE Stampede2

To prepare the evergreen with MPI capability on Stampede2, we need to enable several modules and build netCDF4 with the parallel enabled NetCDF package.

On a login node, the following code prepares the environment.

# Clear all pre-loaded modules
module purge

# Load required modules
module load intel/18.0.2 impi/18.0.2 pnetcdf/1.11.0 python3/3.7.0 phdf5/1.10.4 parallel-netcdf/4.6.2

# Initialize virtual environment with python3
virtualenv -p python3 $HOME/venv

# Start the virtual environment
source $HOME/venv/bin/activate

# Install python modules with no extra setup.
# netCDF4 is installed later separately.
#
pip install pandas numpy pvlib progress Cython mpi4py pyyaml pyinstrument line_profiler tqdm

# Install netCDF4 with extra setup to support parallel I/O
# There are some steps involved.
#

# Download source code
wget https://codeload.github.com/Unidata/netcdf4-python/zip/master -O netCDF4.tar.gz

# Define environment variables
export HDF5_DIR=$TACC_HDF5_DIR
export MPI_INCDIR=$TACC_IMPI_INC

# Manual installation
pip install netCDF4.tar.gz

# Remove the source files
rm netCDF4.tar.gz

On a compute node, the following code tests the environment.

# Start the environment properly
module purge
module load intel/18.0.2 impi/18.0.2 pnetcdf/1.11.0 python3/3.7.0 phdf5/1.10.4 parallel-netcdf/4.6.2
source $HOME/venv/bin/activate

# Run a small test case
ibrun -n 10 python evergreen.py --nc ~/scratch/AnEn/4000_analogs.nc --profile --stations 20

[Weiming-Hu]

UCAR Cheyenne with Intel Compilers and OpenMPI

The steps are similar to the previous script except for the available module names on Cheyenne.

On a login node:

module purge
module load ncarenv intel ncarcompilers openmpi pnetcdf hdf5-mpi python

virtualenv -p python3 $HOME/venv
source $HOME/venv/bin/activate

# yappi was left out and not installed because it didn't work for me.
#
pip install pandas numpy pvlib progress Cython pyyaml pyinstrument line_profiler numba tqdm

# For some reason, mpi4py does not complete with pip installation. Install mpi4py manually.
wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.3.tar.gz
tar -zxf mpi4py-3.0.3.tar.gz
cd mpi4py-3.0.3/
python setup.py build
python setup.py install
cd ..
rm -rf mpi4py-3.0.3.tar.gz mpi4py-3.0.3

# Install netCDF4 manually
wget https://codeload.github.com/Unidata/netcdf4-python/zip/master -O netCDF4.zip
export HDF5_DIR=$HDF5
export MPI_INCDIR=$MPI_ROOT
pip install netCDF4.zip
rm netCDF4.zip

On a compute node:

module purge
module load ncarenv intel ncarcompilers openmpi pnetcdf hdf5-mpi python

source $HOME/venv/bin/activate

mpirun python evergreen.py --nc ~/glade_scratch/AnEn/analogs.nc

[Weiming-Hu]

UCAR Cheyenne with GNU Compilers and MPT

The steps are similar to the previous script except for the available module names on Cheyenne.

On a login node:

module purge
module load gnu mpt pnetcdf hdf5-mpi python

virtualenv -p python3 $HOME/venv
source $HOME/venv/bin/activate

# yappi was left out and not installed because it didn't work for me.
#
pip install pandas numpy pvlib progress Cython pyyaml pyinstrument line_profiler numba tqdm

# For some reason, mpi4py does not complete with pip installation. Install mpi4py manually.
wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.3.tar.gz
tar -zxf mpi4py-3.0.3.tar.gz
cd mpi4py-3.0.3/
python setup.py build
python setup.py install
cd ..
rm -rf mpi4py-3.0.3.tar.gz mpi4py-3.0.3

# Install netCDF4 manually
wget https://codeload.github.com/Unidata/netcdf4-python/zip/master -O netCDF4.zip
export HDF5_DIR=$HDF5
export MPI_INCDIR=$MPI_ROOT
pip install netCDF4.zip
rm netCDF4.zip

On a compute node:

module purge
module load gnu mpt pnetcdf hdf5-mpi python

source $HOME/venv/bin/activate

mpirun python evergreen.py --nc ~/glade_scratch/AnEn/analogs.nc