Building PETSc on LUMI

[hpc4geo]

With a fresh linux environment I first load these modules

module load cray-mpich/8.1.29
module load craype-x86-trento
module load craype-accel-amd-gfx90a
module load rocm/6.0.3

module list reports the following

Currently Loaded Modules:
  1) libfabric/1.15.2.0                         7) lumi-tools/24.05    (S)  13) cray-mpich/8.1.29
  2) craype-network-ofi                         8) init-lumi/0.2       (S)  14) craype-x86-trento
  3) perftools-base/24.03.0                     9) craype/2.7.31.11         15) craype-accel-amd-gfx90a
  4) xpmem/2.8.2-1.0_5.1__g84a27a5.shasta      10) cray-dsmml/0.3.0         16) rocm/6.0.3
  5) cce/17.0.1                                11) cray-libsci/24.03.0
  6) ModuleLabel/label                    (S)  12) PrgEnv-cray/8.5.0

0. PETSc typically found find one of the rocm headers. Hence you will need to set the following environment variable.

export CPATH=$ROCM_PATH/include/rocm-core:$CPATH

1. Building PETSc developer version from the git repo (preferred)

• This page recommends using the dev repo when using GPUs. See Important box at the bottom of the page.
• git clone https://gitlab.com/petsc/petsc.git
• The main branch is the branch to compile.
• I configured using these options

    --download-c2html=0
    --download-cmake
    --download-fblaslapack=1
    --download-hwloc=0
    --download-metis
    --download-parmetis
    --download-sowing=0
    --with-64-bit-indices
    --with-fortran-bindings=0
    --with-hip
    --with-hip-arch=gfx90a
    --with-hipc=hipcc
    --with-memalign=32
    --with-mpi-dir=/opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0
    --with-x=0
    PETSC_ARCH=arch-cray-c-debug-rocm-hip-tandem-vanil

• Notes
  • This is a debug build
  • --download-fblaslapack=1 could be replaced with a more efficient BLAS implementation using something like

    --with-blaslapack-lib='+blas_dir+'/lib/libsci_cray.a '+blas_dir+'/lib/libsci_cray.so '+blas_dir+'/lib/libsci_cray_mpi.a '+blas_dir+'/lib/libsci_cray_mpi.so 

    where blas_dir = /opt/cray/pe/libsci/24.03.0/CRAYCLANG/17.0/x86_64

  • I had issues getting configure to pass with kokkos kernels, e.g. using

    --download-kokkos
    --download-kokkos-kernels

  • My build can be found here /projappl/project_465001082/dmay/software/petsc-dev-git

2. Building PETSc from the release snapshot

• Get the PETSc release

  • From tarball

    wget https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.21.5.tar.gz

  • or from git

    git clone -b release https://gitlab.com/petsc/petsc.git petsc
    git pull # obtain new release fixes (since a prior clone or pull)

• Release 3.21.5 will NOT build with ROCM 6.0.3. To enable compilation a patch must be applied to the PETSc source tree. The patch file PETSc-3.21.5_rocm6.patch is attached to this issue. The patch was found in the EasyBuild distro provided by LUMI (see here /users/maydave2/EasyBuild/SW/LUMI-24.03/L/PETSc/3.21.5-cpeGNU-24.03/easybuild/PETSc-3.21.5_rocm6.patch.

• I configured with these options

  --download-c2html=0
  --download-fblaslapack=1
  --download-hwloc=0
  --download-kokkos
  --download-kokkos-kernels
  --download-metis
  --download-parmetis
  --download-sowing=0
  --with-64-bit-indices
  --with-fortran-bindings=0
  --with-hip
  --with-hip-arch=gfx90a
  --with-hipc=hipcc
  --with-memalign=32
  --with-mpi-dir=/opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0
  --with-x=0
  PETSC_ARCH=arch-cray-c-debug-rocm-hip-tandem-vanil

• Notes

  • This is a debug build.

  • --download-fblaslapack=1 could be replaced with a more efficient BLAS implementation using something like

    --with-blaslapack-lib='+blas_dir+'/lib/libsci_cray.a '+blas_dir+'/lib/libsci_cray.so '+blas_dir+'/lib/libsci_cray_mpi.a '+blas_dir+'/lib/libsci_cray_mpi.so 

    where blas_dir = /opt/cray/pe/libsci/24.03.0/CRAYCLANG/17.0/x86_64

  • My build can be found here /users/maydave2/software/petsc-3.21.5

PETSc-3.21.5_rocm6.patch

[Thomas-Ulrich]

I can confirm I could compile petsc@main with your method, that I sum up hereafter:

module load cray-mpich/8.1.29
module load craype-x86-trento
module load craype-accel-amd-gfx90a
module load rocm/6.0.3
export CPATH=$ROCM_PATH/include/rocm-core:$CPATH
export blas_dir=/opt/cray/pe/libsci/24.03.0/CRAYCLANG/17.0/x86_64
./configure --download-c2html=0 --download-cmake --with-debugging=no  --download-hwloc=0 --download-metis --download-parmetis --download-sowing=0 --with-64-bit-indices --with-fortran-bindings=0 --with-hip --with-hip-arch=gfx90a --with-hipc=hipcc --with-memalign=32 --with-mpi-dir=/opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0 --with-x=0 PETSC_ARCH=arch-cray-c-rocm-hip-tandem-vanille --with-blaslapack-lib="${blas_dir}/lib/libsci_cray.a ${blas_dir}/lib/libsci_cray.so ${blas_dir}/lib/libsci_cray_mpi.a ${blas_dir}/lib/libsci_cray_mpi.so"
make PETSC_DIR=/pfs/lustrep4/scratch/project_465000831/petsc PETSC_ARCH=arch-cray-c-rocm-hip-tandem-vanille all

, and then tandem, with these changes: changes_for_lumi.txt (slightly different from what you posted).

And then:

CC=/opt/rocm-6.0.3/bin/amdclang CXX=/opt/rocm-6.0.3/bin/amdclang++ cmake .. -DCMAKE_PREFIX_PATH=/scratch/project_465000831/petsc/./arch-cray-c-rocm-hip-tandem-vanille/ -DDOMAIN_DIMENSION=3 -DCMAKE_CXX_FLAGS="-I${MPICH_DIR}/include"  -DCMAKE_EXE_LINKER_FLAGS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}"

and then static is working fine. Thank you!