search

geoschem / geos-chem-cloud

Run GEOS-Chem easily on AWS cloud
http://cloud.geos-chem.org
MIT License
39 stars 9 forks source link

[DISCUSSION] A Spack-built software stack for GC-Classic on the AWS cloud #35

Closed yantosca closed 3 years ago

yantosca commented 4 years ago

Overview

I used Spack to build a software stack for GEOS-Chem "Classic" simulations on AWS. I am documenting the steps I took here (at least for now). Maybe later I can make this a README file within the geos-chem-cloud repo.

For some background, see my video tutorial about installing gcc/gfortran 9.2.0 using Spack.

Procedure

1. Log onto AWS

This will start a cloud instance with Intel Skylake architecture.

2. Set up your environment

After logging in to an AWS instance, type these commands:

# Unset env vars
unset NETCDF_HOME
unset NETCDF_FORTRAN_HOME
unset GC_BIN
unset GC_F_BIN
unset GC_INCLUDE
unset GC_F_INCLUDE
unset GC_LIB
unset GC_F_LIB

# Install environment-modules (similar to Lmod modules)
# This will let you use "spack load" to load a module set PATH properly
sudo apt install environment-modules

# Initialize environment-modules
. /etc/profile.d/modules.sh

# Set the path to Spack and run the Spack setup script
export SPACK_ROOT=$HOME/spack
source $SPACK_ROOT/share/spack/setup-env.sh

3. Build libraries with spack

After completing step 2, follow these steps

# Display compiler info.  
# Make sure that gcc@9.2.0 is displayed under linux-ubuntu-skylake_avx512.
spack config get compilers  

# Then type the following commands in this order  (I have added whitespace 
# to better show the different parts of the commands).  
# You can copy & paste these into your terminal window.
#
# Command     module         compiler   options
#------------------------------------------------------------------
spack install netcdf-fortran %gcc@9.2.0
spack install perl           %gcc@9.2.0
spack install flex           %gcc@9.2.0 
spack install cmake          %gcc@9.2.0
spack install gmake          %gcc@9.2.0
spack install gdb            %gcc@9.2.0
spack load    flex
spack load    texinfo
spack install cgdb           %gcc@9.2.0
spack install nco            %gcc@9.2.0
spack install ncview         %gcc@9.2.0 
spack install openjdk        %gcc@9.2.0
spack install tau            %gcc@9.2.0  ~otf2
spack install unzip          %gcc@9.2.0         # Note: Optional
spack load    unzip                             # Note: Optional
spack install emacs          %gcc@9.2.0 +X      # Note: Optional

4. Post-installation configuration

Add this code to your ~/.bashrc file:

# Initialize environment-modules package
if [[ -f /etc/profile.d/modules.sh ]]; then
   source /etc/profile.d/modules.sh
fi

# Initialize spack
export SPACK_ROOT=$HOME/spack
source $HOME/spack/share/spack/setup-env.sh

# Load Spack packages and set relevant environment variables
if [[ -f ~/.init.gc-classic.gfortran92 ]]; then
    source ~/.init.gc-classic.gfortran92
fi

# Alias for loading modules compiled with gfortran 9.2.0
alias load_gf92=`source ~/.init.gc-classic.gfortran92`

And then make sure to copy this file to your ~/init folder and to make it executable:

NOTE: Rename this to .init.gc-classic.gfortran92 and place it in your home directory.

Then when you open a new shell (or type source ~/.bashrc), the modules will be loaded as well.

5. AMI

I used the software stack that I created with Spack to create the following AMI:

Use this to initialize your cloud instance. I will also try to copy this to an S3 bucket.

Also note: this AMI contains 1 day of GEOS-Chem data, but no GEOS-Chem code or UT folders. It is recommended to download the code and to do a dry-run to get the data for whatever period your simulation needs.

6. Further customization

One-time setup:

  1. Fork the repository https://github.com/yantosca/aws-env and clone a local copy.
  2. Replace the name & email fields of aws-env/root/.gitconfig with your name & email: 
    [user]
    name = Bob Yantosca
    email = yantosca@seas.harvard.edu
  3. Also look at the root/.bash_aliases file to add any of your favorite Unix aliases etc.
  4. Push a commit to your local fork of aws-env

Then, once you are in a cloud instance using the GEOSChem-dev-gcc9.2.0 AMI, clone the fork of your aws-env repo to your home directory. Then type:

~/aws-env/initenv.sh
source ~/.bash_aliases

This will apply the settings of your startup scripts.

yuexuyan commented 4 years ago

hi Bob, in your video you picked memory optimised r5.4xlarge, here you chose c5.4xlarge. Which one should be chosen?

yuexuyan commented 4 years ago

on my instance, im having trouble with installing tau by spack. Even with skipping install of 'otf2'.

I just happen to know that ubuntu can install tau with simple 'sodu apt install' command. Is this different from installing tau by spack? I guess the dependent libraries such as 'pdt' are not installed when installing with sodu apt install' command.

and I chosed the same instance with yours (r5.4xlarge) but GEOSChem 12.7.0 tutorial

yantosca commented 4 years ago

You may have better luck with the c5 instances, as they are compute-optimized. Sorry for any confusion.

When you do "sudo apt install" on ubuntu, this installs a pre-built library from a list of libraries that are available. The problem is that you are limited to whatever version and compiler is available. Using Spack, you can build the compiler and then use that compiler to build libraries.

Can you give me a list of commands that you have tried and where they fail?

yuexuyan commented 4 years ago

Thank you very much, I will try c5 later on.

on my r5 instance, I put this line on my terminal after installed other libraries : 'spack install tau %gcc@9.2.0 ~otf2'

and here are the error messages:

==> Error: Failed to install tau due to ChildError: ProcessError: Command exited with status 2: 'make' '-j16' 'install'

9 errors found in build log:

67 grep: ./utils/FixMakefile.info: No such file or directory 68 grep: ./utils/FixMakefile.sed: No such file or directory ... 179 Makefile:16: /tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hoglow4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt: No such file or directory 180 make[1]: No rule to make target '/tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hog low4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt'. S top. ... 183 Makefile:16: /tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hoglow4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt: No such file or directory 184 make[1]: No rule to make target '/tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hog low4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt'. Stop. ... 193 Makefile:16: /tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hoglow4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt: No such file or directory 194 make[1]: No rule to make target '/tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hog low4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt'. S top. ... 197 Makefile:16: /tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hoglow4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt: No such file or directory 198 make[1]: No rule to make target '/tmp/ubuntu/spack-stage/spack-stage-tau-2.29-hoglow4i63yuofns3ocleyh4mdlxxh3z/spack-src/x86_64/lib/Makefile.tau-papi-pthread-pdt'. S top. ... 455 TauBfd.cpp:1020:8: error: 'bfd_get_section_flags' was not declared in this scope; did you mean 'bfd_set_section_flags'? ... 459 TauBfd.cpp:1026:17: error: 'bfd_get_section_vma' was not declared in this scope; did you mean 'bfd_set_section_vma'? ... 463 TauBfd.cpp:1032:24: error: 'bfd_get_section_size' was not declared in this scope; did you mean 'bfd_set_section_size'?

yuexuyan commented 4 years ago

Thank you and by the way,

What is the application you used in video for opening netcdf with ncview. I see a new window popped up from the edge of your screen.

yantosca commented 4 years ago

Sometimes Spack might fail if the internet connection gets interrupted. Then when it tries to build the library it can't find all the files. Usually if you keep trying to rebuild a library it should eventually work.

The netCDF browser is ncview. I had 2 screens on the computer where I made the video. The ncview window popped up in the other screen so I had to manually drag it over.

tejendrapsingh commented 4 years ago

Hi! I am getting an error while building the tau library. Basically, building of libunwind library failed. I am posting the terminal output below.


**`==> Error: ProcessError: Command exited with status 2:**
    **'make' '-j8'**

**==> Warning: Skipping build of tau since libunwind failed**
`
tejendrapsingh commented 4 years ago

Thanks. The issue was resolved here. Cheers!

tejendrapsingh commented 4 years ago

Hi!

So, I have successfully loaded all the requisite library via spack. As detailed in the initial post above, I have copied .init.gc-classic.gfortran92 in my home directory after editing the ~/.bashrc file appropriately. Then, when I am doing source ~/.bashrc, I am getting the following error. Any help is appreciated. Thanks. 


: command not found...
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifierUDE
': not a valid identifier
': not a valid identifierRAN_BIN
': not a valid identifierRAN_HOME
': not a valid identifierRAN_INCLUDE
': not a valid identifierRAN_LIB
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifier
': not a valid identifierE
': not a valid identifier
': not a valid identifierEADS
': not a valid identifierZE
: command not found...
: command not found...
: command not found...
: command not found...
bash: /home/tejendrapsingh/.init.gc-classic.gfortran92: line 78: syntax error near unexpected token `$'do\r''
bash: /home/tejendrapsingh/.init.gc-classic.gfortran92: line 78: `for f in ${pkg'[@]}; do

My init file: init.gc-classic.gfortran92.txt

yantosca commented 4 years ago

You might have a syntax error in your .bashrc.

yantosca commented 3 years ago

Also, it is often useful to build the Spack libraries with the target=x86_64 option. This builds for generic x86-64 chipsets. Use this if your computer cluster has a number of different nodes with different chip types.

# Then type the following commands in this order  (I have added whitespace 
# to better show the different parts of the commands).  
# You can copy & paste these into your terminal window.
#
# Command     module         compiler   options
#------------------------------------------------------------------
spack install netcdf-fortran %gcc@9.2.0 target=x86_64 
spack install perl           %gcc@9.2.0 target=x86_64 
spack install flex           %gcc@9.2.0 target=x86_64 
spack install cmake          %gcc@9.2.0 target=x86_64 
spack install gmake          %gcc@9.2.0 target=x86_64 
spack install gdb            %gcc@9.2.0 target=x86_64 
spack load    flex
spack load    texinfo
spack install cgdb           %gcc@9.2.0 target=x86_64 
spack install nco            %gcc@9.2.0 target=x86_64 
spack install ncview         %gcc@9.2.0 target=x86_64 
spack install openjdk        %gcc@9.2.0 target=x86_64 
spack install tau            %gcc@9.2.0  ~otf2 target=x86_64 
spack install unzip          %gcc@9.2.0  target=x86_64       # Note: Optional
spack load    unzip                                          # Note: Optional
spack install emacs          %gcc@9.2.0 +X  target=x86_64    # Note: Optional
yantosca commented 3 years ago

This discussion is now superseded by https://github.com/geoschem/geos-chem/issues/637.