mfem build error with +rocm

[balay]

/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/hip-5.5.1-36grozs3lkqmnph77fzw7tfbykoccwci/bin/hipcc -fPIC  -O3 -std=c++14 --offload-arch=gfx90a -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/hypre-2.\
30.0-sptx5zqbwmsaq7ga6kzptto6yshntpt3/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/openmpi-4.1.6-726wgocxnrdcpvhnyctkrwk5brwxpble/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/su\
perlu-dist-8.2.0-b7wlokhclgiyo5m3cs5bxjqbskmoyoyz/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/parmetis-4.0.3-357lo65nzpdjosuzjoqnpt2scrwu3jvf/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/\
gcc-11.4.0/metis-5.1.0-rv7yvjaonbvxlxr5pqvrgtqjlmrunnlr/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/sundials-6.6.1-r3237tcbjg3wnfg7q5qxzoje4ewgazbw/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04\
-zen4/gcc-11.4.0/petsc-3.20.0-rg55eg5xkqecpv2ed3l2tpdxxswlqu5o/include -I/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/hipsparse-5.5.1-hrdhigl6dssav4iifhwfzc42wfctyvjm/include -I/scratch/balay/spack/opt/spack/linux-ubu\
ntu22.04-zen4/gcc-11.4.0/zlib-ng-2.1.3-g3q24jtcwja4hbzlaqj5xpfppsd4kkgc/include  -c mesh/segment.cpp -o mesh/segment.o
In file included from linalg/handle.cpp:15:
In file included from linalg/petsc.hpp:45:
/scratch/balay/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-11.4.0/petsc-3.20.0-rg55eg5xkqecpv2ed3l2tpdxxswlqu5o/include/petscsystypes.h:531:16: fatal error: 'thrust/complex.h' file not found
      #include <thrust/complex.h>
               ^~~~~~~~~~~~~~~~~~

spack-build-out.txt

[balay]

for now - using:

diff --git a/var/spack/repos/builtin/packages/mfem/package.py b/var/spack/repos/builtin/packages/mfem/package.py
index 5fac0860ea..6e14053355 100644
--- a/var/spack/repos/builtin/packages/mfem/package.py
+++ b/var/spack/repos/builtin/packages/mfem/package.py
@@ -258,6 +258,7 @@ class Mfem(Package, CudaPackage, ROCmPackage):

     depends_on("mpi", when="+mpi")
     depends_on("hipsparse", when="@4.4.0:+rocm")
+    depends_on("rocthrust", when="@4.4.0:+rocm")
     depends_on("hypre@2.10.0:2.13", when="@:3.3+mpi")
     depends_on("hypre@:2.20.0", when="@3.4:4.2+mpi")
     depends_on("hypre@:2.23.0", when="@4.3.0+mpi")
@@ -933,7 +929,8 @@ def find_optional_library(name, prefix):
             #   hip_libs += find_libraries("libamdhip64", spec["hip"].prefix.lib)
             if "^hipsparse" in spec:  # hipsparse is needed @4.4.0:+rocm
                 hipsparse = spec["hipsparse"]
-                options += ["HIP_OPT=%s" % hipsparse.headers.cpp_flags]
+                rocthrust = spec["rocthrust"]
+                options += ["HIP_OPT=%s" % hipsparse.headers.cpp_flags +' '+ rocthrust.headers.cpp_flags]
                 hip_libs += hipsparse.libs
                 # Note: MFEM's defaults.mk wants to find librocsparse.* in
                 # $(HIP_DIR)/lib, so we set HIP_DIR to be $ROCM_PATH when using

[v-dobrev]

It looks like rocthrust is needed by PETSc. This probably means that PETSc does not "export" the rocthrust path in its petscvariables file -- otherwise MFEM will pick it up from there (from the variable PETSC_CC_INCLUDES). Maybe there is a different variable that MFEM needs to read when PETSc is built with HIP?

[v-dobrev]

Actually, it looks like PETSc is built +shared in Spack, so the MFEM Spack package configures MFEM to use the headers property of the petsc package (and does not use the petscvariables file). See https://github.com/spack/spack/blob/8c257d55b49857179c10234d2b04782ce1f5c1ff/var/spack/repos/builtin/packages/mfem/package.py#L813-L821.

It may be best to make sure PETSc's Spack package exports the rocthrust path in its headers property when needed. If you don't like this solution, we'll have to add it in the MFEM Spack package.

[balay]

Hm - this is used by hip sources - and the corresponding flag for use with hipcc is HIPC_FLAGS - but yeah - I don't see it in pkg config file

[v-dobrev]

@balay, can you try your HIP build with https://github.com/spack/spack/pull/40768 instead of the patch you posted above? Thanks!

[balay]

@v-dobrev thanks! I see its merged to spack/develop. I've rebased xsdk-1.0.0 branch - and removed my patch.

This build (with hip) does work for me .

[v-dobrev]

Great! Closing the issue then.