search

xsdk-project / xsdk-examples

Example codes demonstrating the use of various XSDK packages in combination.
17 stars 10 forks source link

Testing of release v0.4.0 #40

Closed v-dobrev closed 1 year ago

v-dobrev commented 1 year ago

Hello @xsdk-project/ecp-xsdk,

Let's use this issue to coordinate the testing of the v0.4.0 release.

Summary

I propose we perform the testing using the following branches:

Instructions for testing with CMake

Mac

 #### Install xSDK packages via Spack on Mac: ```console ./bin/spack install --no-cache --fresh \ xsdk@0.8.0%apple-clang~alquimia~arborx+precice \ ^dealii~oce~hdf5 \ ^mfem+ginkgo+hiop+pumi+strumpack ``` Notes about this spec: * The compiler configuration where this was tested is `apple-clang@14.0.3` with `gfortran@13.1.0` on Apple silicon Mac. * Building with `alquimia` fails, so it is disabled. * Disable `arborx` because it has no xSDK examples and it brings in `kokkos` as dependency which conflicts with the built-in kokkos in `trilinos`. * `precice` is disabled by default, however we can enable it and test the deal.II + preCICE example. * Disable the `oce` variant of deal.II because building `oce` takes time and we don't need it. * Disable the `hdf5` variant of deal.II because it has linking problems, see https://github.com/xsdk-project/xsdk-examples/pull/36#issuecomment-1540927535. * Enable the following MFEM variants which are needed by the respective xSDK examples: `+ginkgo+hiop+pumi+strumpack`. * This spec does not build `amrex` because it is not enabled in the `xsdk` package when using the `apple-clang` compiler. Therefore, the AMReX + SUNDIALS xSDK example will not be tested. * This spec does not build `plasma` because it is enabled in the `xsdk` package only for `gcc` v6 or newer. Therefore, the PLASMA xSDK example will not be tested. #### Using the xSDK packages installed via Spack on Mac: ```console SPACK_DIR= cd $SPACK_DIR ./bin/spack view add xsdk-0.8.0 -i xsdk@0.8.0 cd mkdir build cd build cmake .. \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0 \ -DCMAKE_VERBOSE_MAKEFILE=ON \ -DCMAKE_Fortran_COMPILER=mpifort \ -DENABLE_CUDA=OFF \ -DENABLE_HIP=OFF \ -DENABLE_AMREX=OFF \ -DENABLE_DEAL_II=ON \ -DENABLE_GINKGO=ON \ -DENABLE_HEFFTE=ON \ -DENABLE_HIOP=ON \ -DENABLE_HYPRE=ON \ -DENABLE_MAGMA=OFF \ -DENABLE_MFEM=ON \ -DENABLE_PETSC=ON \ -DENABLE_PLASMA=OFF \ -DENABLE_PRECICE=ON \ -DENABLE_PUMI=ON \ -DENABLE_STRUMPACK=ON \ -DENABLE_SUNDIALS=ON \ -DENABLE_SUPERLU=ON \ -DENABLE_TASMANIAN=ON \ -DENABLE_TRILINOS=ON make -j ctest --output-on-failure ```

CPU build on Tioga (LLNL)

 #### Install xSDK packages (CPU-only build) via Spack on Tioga:
Tioga compilers.yaml ```yaml compilers: - compiler: spec: cce@15.0.1 paths: cc: cc cxx: CC f77: ftn fc: ftn operating_system: rhel8 target: any modules: # TCE Standard Environment - StdEnv # PrgEnv-cray loads: craype, cray-mpich, cray-libsci (default versions) # craype loads: (none) # cray-mpich loads: (none) # cray-libsci loads: (none) - PrgEnv-cray # for consistency load specific versions in case the defaults change - craype/2.7.20 - cray-mpich/8.1.25 - cray-libsci/23.02.1.1 # cce/15.0.1 loads: (none) - cce/15.0.1 - craype-x86-trento extra_rpaths: [] - compiler: spec: gcc@12.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: rhel8 target: any modules: - StdEnv # PrgEnv-gnu loads: gcc, craype, cray-mpich, cray-libsci (default versions) # gcc loads: PrgEnv-gnu # craype loads: (none) # cray-mpich loads: (none) # cray-libsci loads: (none) - PrgEnv-gnu # for consistency load specific versions in case the defaults change - gcc/12.2.0 - craype/2.7.20 - cray-mpich/8.1.25 - cray-libsci/23.02.1.1 - craype-x86-trento extra_rpaths: [] ```
Tioga packages.yaml ```yaml packages: autoconf: externals: - spec: autoconf@2.69 prefix: /usr automake: externals: - spec: automake@1.16.1 prefix: /usr bison: externals: - spec: bison@3.0.4 prefix: /usr bzip2: externals: - spec: bzip2@1.0.6 prefix: /usr cmake: externals: - spec: cmake@3.24.2 modules: - cmake/3.24.2 - spec: cmake@3.22.4 modules: - cmake/3.22.4 - spec: cmake@3.19.2 modules: - cmake/3.19.2 buildable: false cpio: externals: - spec: cpio@2.12 prefix: /usr diffutils: externals: - spec: diffutils@3.6 prefix: /usr expat: externals: - spec: expat@2.2.5 prefix: /usr file: externals: - spec: file@5.33 prefix: /usr findutils: externals: - spec: findutils@4.6.0 prefix: /usr flex: externals: - spec: flex@2.6.1+lex prefix: /usr fontconfig: externals: - spec: fontconfig@2.13.1 prefix: /usr freetype: externals: - spec: freetype@2.9.1 prefix: /usr gawk: externals: - spec: gawk@4.2.1 prefix: /usr gettext: externals: - spec: gettext@0.19.8.1 prefix: /usr ghostscript: externals: - spec: ghostscript@9.27 prefix: /usr git: externals: - spec: git@2.31.1 prefix: /usr gmake: externals: - spec: gmake@4.2.1 prefix: /usr gnutls: externals: - spec: gnutls@3.6.16 prefix: /usr groff: externals: - spec: groff@1.22.3 prefix: /usr hwloc: externals: - spec: hwloc@2.8.0 prefix: /usr libpng: externals: - spec: libpng@1.6.34 prefix: /usr libtool: externals: - spec: libtool@2.4.6 prefix: /usr libx11: externals: - spec: libx11@1.6.8 prefix: /usr libxcb: externals: - spec: libxcb@1.13.1 prefix: /usr libxml2: externals: - spec: libxml2@2.9.7 prefix: /usr libxext: externals: - spec: libxext@1.3.4 prefix: /usr libxt: externals: - spec: libxt@1.1.5 prefix: /usr m4: externals: - spec: m4@1.4.18 prefix: /usr ncurses: externals: - spec: ncurses@6.1 prefix: /usr opengl: externals: - spec: opengl@4.5 prefix: /usr openglu: externals: - spec: openglu@1.3 prefix: /usr openssh: externals: - spec: openssh@8.0p1 prefix: /usr openssl: externals: - spec: openssl@1.1.1k prefix: /usr perl: externals: - spec: perl@5.26.3 prefix: /usr rsync: externals: - spec: rsync@3.1.3 prefix: /usr ruby: externals: - spec: ruby@2.5.9 prefix: /usr sed: externals: - spec: sed@4.5 prefix: /usr tar: externals: - spec: tar@1.30 prefix: /usr texinfo: externals: - spec: texinfo@6.5 prefix: /usr xproto: externals: # see /usr/share/pkgconfig/xproto.pc - spec: xproto@7.0.32 prefix: /usr xrandr: externals: # xrandr --version - spec: xrandr@1.5.0 prefix: /usr xxd-standalone: externals: # vim-common version - spec: xxd-standalone@8.0.1763 prefix: /usr xz: externals: - spec: xz@5.2.4 prefix: /usr # ROCm 5.4.3 hip: externals: - spec: hip@5.4.3 prefix: /opt/rocm-5.4.3 extra_attributes: compilers: c: /opt/rocm-5.4.3/llvm/bin/clang c++: /opt/rocm-5.4.3/llvm/bin/clang++ hip: /opt/rocm-5.4.3/hip/bin/hipcc buildable: false hsa-rocr-dev: externals: - spec: hsa-rocr-dev@5.4.3 prefix: /opt/rocm-5.4.3 extra_attributes: compilers: c: /opt/rocm-5.4.3/llvm/bin/clang cxx: /opt/rocm-5.4.3/llvm/bin/clang++ buildable: false llvm-amdgpu: externals: - spec: llvm-amdgpu@5.4.3 prefix: /opt/rocm-5.4.3/llvm extra_attributes: compilers: c: /opt/rocm-5.4.3/llvm/bin/clang cxx: /opt/rocm-5.4.3/llvm/bin/clang++ buildable: false hipcub: externals: - spec: hipcub@5.4.3 prefix: /opt/rocm-5.4.3/hipcub hipblas: externals: - spec: hipblas@5.4.3 prefix: /opt/rocm-5.4.3/hipblas hipsolver: externals: - spec: hipsolver@5.4.3 prefix: /opt/rocm-5.4.3/hipsolver rocblas: externals: - spec: rocblas@5.4.3 prefix: /opt/rocm-5.4.3/rocblas rocsolver: externals: - spec: rocsolver@5.4.3 prefix: /opt/rocm-5.4.3/rocsolver hipsparse: externals: - spec: hipsparse@5.4.3 prefix: /opt/rocm-5.4.3/hipsparse rocsparse: externals: - spec: rocsparse@5.4.3 prefix: /opt/rocm-5.4.3/rocsparse rocthrust: externals: - spec: rocthrust@5.4.3 prefix: /opt/rocm-5.4.3/rocthrust rocrand: externals: - spec: rocrand@5.4.3 prefix: /opt/rocm-5.4.3/rocrand rocprim: externals: - spec: rocprim@5.4.3 prefix: /opt/rocm-5.4.3/rocprim rocfft: externals: - spec: rocfft@5.4.3 prefix: /opt/rocm-5.4.3/rocfft cray-mpich: externals: - spec: "cray-mpich@8.1.25" modules: - cray-mpich/8.1.25 - craype-network-ofi buildable: false cray-libsci: buildable: false # version: [23.02.1.1] externals: # - spec: cray-libsci@23.02.1.1 # modules: # - cray-libsci/23.02.1.1 - spec: "cray-libsci@23.02.1.1+mpi" modules: - cray-libsci/23.02.1.1 - cray-mpich/8.1.25 - craype-network-ofi all: providers: mpi: [cray-mpich] lapack: [cray-libsci] blas: [cray-libsci] # scalapack: [netlib-scalapack] scalapack: [cray-libsci+mpi] libglx: [opengl] glu: [openglu] xxd: [xxd-standalone] compiler: [cce@15.0.1] ```
```console ./bin/spack install -j 96 --no-cache --fresh \ xsdk@0.8.0%gcc~arborx \ ^dealii~oce~threads~hdf5 cxxstd=14 \ ^mfem+ginkgo+hiop+pumi+strumpack \ ^openssl@1.1.1k ``` #### Using the xSDK packages (CPU-only build) installed via Spack on Tioga: ```console SPACK_DIR= cd $SPACK_DIR ./bin/spack view add xsdk-0.8.0 -i xsdk@0.8.0~rocm ml PrgEnv-gnu cd mkdir build cd build cmake .. \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -DCMAKE_Fortran_COMPILER=gfortran \ -DMPIEXEC_EXECUTABLE="flux;run" \ -DMPIEXEC_NUMPROC_FLAG="-n" \ -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0 \ -DCMAKE_VERBOSE_MAKEFILE=ON \ -DENABLE_CUDA=OFF \ -DENABLE_HIP=OFF \ -DENABLE_AMREX=ON \ -DENABLE_DEAL_II=ON \ -DENABLE_GINKGO=ON \ -DENABLE_HEFFTE=ON \ -DENABLE_HIOP=ON \ -DENABLE_HYPRE=ON \ -DENABLE_MAGMA=OFF \ -DENABLE_MFEM=ON \ -DENABLE_PETSC=ON \ -DENABLE_PLASMA=ON \ -DENABLE_PUMI=ON \ -DENABLE_STRUMPACK=ON \ -DENABLE_SUNDIALS=ON \ -DENABLE_SUPERLU=ON \ -DENABLE_TASMANIAN=ON \ -DENABLE_TRILINOS=ON make -j ctest --output-on-failure ``` Note: the test `STRUMPACK-sparse` may fail, however it actually works when run with the command ```console cd /build/strumpack flux run -n 4 env OMP_NUM_THREADS=4 ./sparse 24 --sp_compression hodlr --hodlr_butterfly_levels 10 ```

HIP build on Tioga (LLNL)

 #### Install xSDK packages (`+rocm` build) via Spack on Tioga: Tioga `compilers.yaml` and `packages.yaml`: same as in the section "CPU build on Tioga (LLNL)" above. ```console ./bin/spack install -j 96 --no-cache --fresh \ xsdk@0.8.0%gcc~arborx~phist +rocm amdgpu_target=gfx90a \ ^dealii~oce~threads~hdf5 cxxstd=14 \ ^mfem+ginkgo+hiop+pumi+strumpack \ ^strumpack~slate \ ^openssl@1.1.1k ``` #### Using the xSDK packages (`+rocm` build) installed via Spack on Tioga: ```console SPACK_DIR= cd $SPACK_DIR ./bin/spack view add xsdk-0.8.0-rocm -i xsdk@0.8.0+rocm ml PrgEnv-gnu ml rocm/5.4.3 cd mkdir build cd build cmake .. \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -DCMAKE_Fortran_COMPILER=gfortran \ -DCMAKE_HIP_ARCHITECTURES="gfx90a" \ -DMPIEXEC_EXECUTABLE="flux;run" \ -DMPIEXEC_NUMPROC_FLAG="-n" \ -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0-rocm \ -DCMAKE_VERBOSE_MAKEFILE=ON \ -DENABLE_CUDA=OFF \ -DENABLE_HIP=ON \ -DENABLE_AMREX=ON \ -DENABLE_DEAL_II=ON \ -DENABLE_GINKGO=ON \ -DENABLE_HEFFTE=ON \ -DENABLE_HIOP=ON \ -DENABLE_HYPRE=ON \ -DENABLE_MAGMA=ON \ -DENABLE_MFEM=ON \ -DENABLE_PETSC=ON \ -DENABLE_PLASMA=ON \ -DENABLE_PUMI=ON \ -DENABLE_STRUMPACK=ON \ -DENABLE_SUNDIALS=ON \ -DENABLE_SUPERLU=ON \ -DENABLE_TASMANIAN=ON \ -DENABLE_TRILINOS=ON make -j ctest --output-on-failure ```

Machines and configurations to test with CMake

Instructions for testing with Spack

(TBD)

Spack machine configurations

v-dobrev commented 1 year ago

I have merged the following branches/PRs in testing-v0.4.0:

Please feel free to merge/re-merge branches as needed and update the above list.

balay commented 1 year ago

I've updated the 'spack test' section instructions.