search

ECP-WarpX / impactx

high-performance modeling of beam dynamics in particle accelerators with collective effects
https://impactx.readthedocs.io
Other
26 stars 20 forks source link

Profiling ImpactX on Crusher with AMD tools #310

Open n01r opened 1 year ago

n01r commented 1 year ago

During the OLCF crusher hackathon I had tried to build/run ImpactX for profiling. I tried a CPU-only version first to then go ahead and use AMD-provided tools on Crusher for profiling of the host side. However, running failed and I am trying again today to document the status of where it is failing.

To reproduce, first load modules for the dependencies (starting off here from the one for Crusher from the WarpX repository since ImpactX and WarpX share the same dependencies). https://warpx.readthedocs.io/en/latest/install/hpc/crusher.html#id1 For convenience, download and save under $HOME/crusher_impactx.profile.

Commands used to build the most recent development branch

source $HOME/crusher_impactx.profile

mkdir -p $HOME/src
# download defaults to development branch
git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx
cd $HOME/src/impactx

cmake -S . -B build -DImpactX_COMPUTE=NOACC -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j 16

Running the FODO example works with this. However, using Omnitrace on this example did not (back during last year)

On the other hand, building for HIP did not work and the install failed

rm -rf build

cmake -S . -B build -DImpactX_COMPUTE=HIP -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j 16

See the build and error output by expanding the following details section

``` [mgarten@login2.crusher impactx]$ cmake -S . -B build -DImpactX_COMPUTE=NOACC -DCMAKE_BUILD_TYPE=RelWithDebInfo -- Cray Programming Environment 2.7.17 C -- Cray Programming Environment 2.7.17 CXX -- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache -- Downloading ABLASTR ... -- ABLASTR repository: https://github.com/ECP-WarpX/WarpX.git (23.01) -- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache -- Downloading AMReX ... -- AMReX repository: https://github.com/AMReX-Codes/amrex.git (23.01) -- CMake version: 3.23.2 -- AMReX installation directory: /usr/local -- Build type set by user to 'RelWithDebInfo'. -- Building AMReX with AMReX_SPACEDIM = 3 -- Configuring AMReX with the following options enabled: -- AMReX_BUILD_SHARED_LIBS -- AMReX_PRECISION = DOUBLE -- AMReX_MPI -- AMReX_MPI_THREAD_MULTIPLE -- AMReX_LINEAR_SOLVERS -- AMReX_PARTICLES -- AMReX_PARTICLES_PRECISION = DOUBLE -- AMReX_PIC -- AMReX_TINY_PROFILE -- AMReX configuration summary: -- Build type = RelWithDebInfo -- Install directory = /usr/local -- C++ compiler = /opt/cray/pe/craype/2.7.17/bin/CC -- C++ defines = -- C++ flags = -O2 -g -DNDEBUG -I/opt/rocm-5.2.0/include -- C++ include paths = -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -- Link line = -- AMReX: Using version '23.01' (23.01) WarpX build configuration: Version: 23.01 (23.01) C++ Compiler: Clang 14.0.6 CrayPrgEnv /opt/cray/pe/craype/2.7.17/bin/CC Installation prefix: /usr/local bin: bin lib: lib64 include: include cmake: lib64/cmake/WarpX Build type: RelWithDebInfo Build options: APP: OFF ASCENT: OFF COMPUTE: NOACC DIMS: 3 Embedded Boundary: OFF GPU clock timers: OFF IPO/LTO: OFF LIB: OFF MPI: ON PSATD: OFF PRECISION: DOUBLE PARTICLE PRECISION: DOUBLE OPENPMD: OFF QED: OFF QED table generation: OFF SENSEI: OFF -- ABLASTR: Using version '' (23.01) -- Downloading pybind11 ... -- pybind11 repository: https://github.com/pybind/pybind11.git (v2.10.1) -- pybind11 v2.10.1 -- Downloading pyAMReX ... -- pyAMReX repository: https://github.com/AMReX-Codes/pyamrex.git (f3ba017531e2d1569e63428919632cf908435e56) -- AMReX::amrex target already imported -- pybind11::module target already imported pyAMReX build configuration: Version: 23.01 C++ Compiler: Clang 14.0.6 CrayPrgEnv /opt/cray/pe/craype/2.7.17/bin/CC Installation prefix: /usr/local bin: bin lib: lib64 include: include cmake: lib64/cmake/pyAMReX python: lib64/python3.9/site-packages Build type: RelWithDebInfo ImpactX build configuration: Version: 23.02 (23.02-1-g1a134a03ef79) C++ Compiler: Clang 14.0.6 CrayPrgEnv /opt/cray/pe/craype/2.7.17/bin/CC Installation prefix: /usr/local bin: bin lib: lib64 include: include cmake: lib64/cmake/ImpactX python: lib64/python3.9/site-packages Build type: RelWithDebInfo Build options: APP: ON COMPUTE: NOACC IPO/LTO: OFF LIB: ON (static) MPI: ON PRECISION: DOUBLE PYTHON: ON OPENPMD: OFF -- Configuring done -- Generating done -- Build files have been written to: /ccs/home/mgarten/src/impactx/build [mgarten@login2.crusher impactx]$ rm -rf build [mgarten@login2.crusher impactx]$ cmake -S . -B build -DImpactX_COMPUTE=HIP -DCMAKE_BUILD_TYPE=RelWithDebInfo -- The C compiler identification is Clang 14.0.6 -- The CXX compiler identification is Clang 14.0.6 -- Cray Programming Environment 2.7.17 C -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /opt/cray/pe/craype/2.7.17/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Cray Programming Environment 2.7.17 CXX -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /opt/cray/pe/craype/2.7.17/bin/CC - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache -- Downloading ABLASTR ... -- ABLASTR repository: https://github.com/ECP-WarpX/WarpX.git (23.02) -- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache -- Downloading AMReX ... -- AMReX repository: https://github.com/AMReX-Codes/amrex.git (23.02) -- CMake version: 3.23.2 -- AMReX installation directory: /usr/local -- Build type set by user to 'RelWithDebInfo'. -- Building AMReX with AMReX_SPACEDIM = 3 -- Configuring AMReX with the following options enabled: -- AMReX_PRECISION = DOUBLE -- AMReX_GPU_BACKEND = HIP -- AMReX_MPI -- AMReX_MPI_THREAD_MULTIPLE -- AMReX_LINEAR_SOLVERS -- AMReX_PARTICLES -- AMReX_PARTICLES_PRECISION = DOUBLE -- AMReX_TINY_PROFILE -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- Found MPI_C: /opt/cray/pe/craype/2.7.17/bin/cc (found version "3.1") -- Found MPI_CXX: /opt/cray/pe/craype/2.7.17/bin/CC (found version "3.1") -- Found MPI: TRUE (found version "3.1") found components: C CXX -- hip::amdhip64 is SHARED_LIBRARY -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Failed -- Found HIP: -- HIP: Runtime=rocclr Compiler=clang Path=/opt/rocm-5.2.0/hip -- hip::amdhip64 is SHARED_LIBRARY -- hip::amdhip64 is SHARED_LIBRARY -- hip::amdhip64 is SHARED_LIBRARY -- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD -- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD - Failed -- AMReX configuration summary: -- Build type = RelWithDebInfo -- Install directory = /usr/local -- C++ compiler = /opt/cray/pe/craype/2.7.17/bin/CC -- C++ defines = -D__HIP_PLATFORM_HCC__=1 -D__HIP_PLATFORM_AMD__=1 -- C++ flags = -O2 -g -DNDEBUG -I/opt/rocm-5.2.0/include -mllvm -amdgpu-early-inline-all=true -amdgpu-function-calls=false -m64 -munsafe-fp-atomics -x hip --offload-arch=gfx90a -- C++ include paths = -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I/opt/rocm-5.2.0/hip/include -I/opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -I/opt/rocm-5.2.0/include -I/opt/rocm-5.2.0/include/hiprand -I/opt/rocm-5.2.0/include/rocrand -I/opt/rocm-5.2.0/include/ -- Link line = /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a -- AMReX: Using version '23.02' (23.02) -- Found Git: /usr/bin/git (found version "2.35.3") WarpX build configuration: Version: 23.02 (23.02) C++ Compiler: Clang 14.0.6 CrayPrgEnv /opt/cray/pe/craype/2.7.17/bin/CC Installation prefix: /usr/local bin: bin lib: lib64 include: include cmake: lib64/cmake/WarpX Build type: RelWithDebInfo Build options: APP: OFF ASCENT: OFF COMPUTE: HIP DIMS: 3 Embedded Boundary: OFF GPU clock timers: ON IPO/LTO: OFF LIB: OFF MPI: ON PSATD: OFF PRECISION: DOUBLE PARTICLE PRECISION: DOUBLE OPENPMD: OFF QED: OFF QED table generation: OFF SENSEI: OFF -- ABLASTR: Using version '' (23.02) ImpactX build configuration: Version: 23.02 (23.02-1-g1a134a03ef79) C++ Compiler: Clang 14.0.6 CrayPrgEnv /opt/cray/pe/craype/2.7.17/bin/CC Installation prefix: /usr/local bin: bin lib: lib64 include: include cmake: lib64/cmake/ImpactX Build type: RelWithDebInfo Build options: APP: ON COMPUTE: HIP IPO/LTO: OFF LIB: OFF MPI: ON PRECISION: DOUBLE PYTHON: OFF OPENPMD: OFF -- Configuring done -- Generating done -- Build files have been written to: /ccs/home/mgarten/src/impactx/build [mgarten@login2.crusher impactx]$ cmake --build build -j 16 [ 0%] Building CXX object CMakeFiles/buildInfoapp.dir/app/AMReX_buildInfo.cpp.o [ 0%] Building CXX object _deps/fetchedablastr-build/CMakeFiles/buildInfoablastr.dir/ablastr/AMReX_buildInfo.cpp.o [ 1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BlockMutex.cpp.o [ 1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX.cpp.o [ 1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_error_fi.cpp.o [ 3%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Version.cpp.o [ 5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParmParse.cpp.o [ 5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_parmparse_fi.cpp.o [ 5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Utility.cpp.o [ 5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FileSystem.cpp.o [ 7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_DistributionMapping.cpp.o [ 7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Random.cpp.o [ 7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParallelDescriptor.cpp.o [ 9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParallelContext.cpp.o [ 9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ForkJoin.cpp.o [ 9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_VisMF.cpp.o [ 11%] Linking CXX static library lib/libbuildInfoapp.a [ 11%] Built target buildInfoapp [ 13%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_AsyncOut.cpp.o [ 15%] Linking CXX static library ../../lib/libbuildInfoablastr.a [ 15%] Built target buildInfoablastr [ 15%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BackgroundThread.cpp.o [ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Arena.cpp.o [ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BArena.cpp.o [ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_CArena.cpp.o [ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PArena.cpp.o [ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BLBackTrace.cpp.o [ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_NFiles.cpp.o [ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_parstream.cpp.o [ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FabConv.cpp.o [ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FPC.cpp.o [ 23%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_VectorIO.cpp.o [ 23%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IntConv.cpp.o [ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Box.cpp.o [ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxIterator.cpp.o [ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IntVect.cpp.o [ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IndexType.cpp.o [ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Orientation.cpp.o [ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Periodicity.cpp.o [ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_RealBox.cpp.o [ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_RealVect.cpp.o [ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxList.cpp.o [ 30%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxArray.cpp.o [ 30%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxDomain.cpp.o [ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FArrayBox.cpp.o [ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IArrayBox.cpp.o [ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BaseFab.cpp.o [ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MultiFab.cpp.o [ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MFCopyDescriptor.cpp.o [ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_iMultiFab.cpp.o [ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FabArrayBase.cpp.o [ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MFIter.cpp.o [ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_CoordSys.cpp.o [ 38%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Geometry.cpp.o [ 38%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MultiFabUtil.cpp.o [ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BCRec.cpp.o [ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PhysBCFunct.cpp.o [ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BCUtil.cpp.o [ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilCC_C.cpp.o [ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilFC_C.cpp.o [ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilND_C.cpp.o [ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_NonLocalBC.cpp.o [ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PlotFileUtil.cpp.o [ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PlotFileDataImpl.cpp.o [ 46%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuControl.cpp.o [ 46%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuDevice.cpp.o [ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuUtility.cpp.o [ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuAsyncArray.cpp.o [ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuElixir.cpp.o [ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Machine.cpp.o [ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MemPool.cpp.o [ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser.cpp.o [ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser_Exe.cpp.o [ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser_Y.cpp.o [ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_parser.lex.cpp.o [ 53%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_parser.tab.cpp.o [ 53%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser.cpp.o [ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser_Exe.cpp.o [ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser_Y.cpp.o [ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_iparser.lex.cpp.o [ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_iparser.tab.cpp.o [ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BLProfiler.cpp.o [ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_TinyProfiler.cpp.o [ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MPMD.cpp.o [ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_Mask.cpp.o [ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_MultiMask.cpp.o [ 61%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_LO_BCTYPES.cpp.o [ 61%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_AmrCore.cpp.o [ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_Cluster.cpp.o [ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_ErrorList.cpp.o [ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_FillPatchUtil.cpp.o [ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_FluxRegister.cpp.o [ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_InterpBase.cpp.o [ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_MFInterpolater.cpp.o [ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_Interpolater.cpp.o [ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_TagBox.cpp.o [ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_AmrMesh.cpp.o [ 69%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_InterpFaceRegister.cpp.o [ 69%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLMG.cpp.o [ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp.o [ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o [ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o [ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp.o [ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp.o [ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp.o [ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o [ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp.o [ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp.o [ 76%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/OpenBC/AMReX_OpenBC.cpp.o [ 76%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_TracerParticles.cpp.o [ 78%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleMPIUtil.cpp.o [ 78%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleUtil.cpp.o clang-14: /home/jenkins/llvm/lib/Target/AMDGPU/SIOptimizeVGPRLiveRange.cpp:497: void {anonymous}::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector&) const: Assertion `UseMI->isPHI() && "Uses should be PHI in Endif block"' failed. PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -flocal-restrict -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name AMReX_MLNodeLaplacian_sync.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -resource-dir /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6 -dependency-file CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o.d -MT _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include/cuda_wrappers -idirafter /opt/rocm-5.2.0/include -include __clang_hip_runtime_wrapper.h -isystem /opt/rocm-5.2.0/hip/include -isystem /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -isystem /opt/rocm-5.2.0/include/hiprand -isystem /opt/rocm-5.2.0/include/rocrand -isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -isystem /opt/cray/pe/cce/14.0.2/cce/x86_64/include/craylibs -D __CRAY_X86_TRENTO -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__=1 -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build -I /opt/rocm-5.2.0/include -D NDEBUG -I /opt/cray/pe/mpich/8.1.18/ofi/cray/10.0/include -I /opt/cray/pe/libsci/22.08.1.1/CRAY/9.0/x86_64/include -I /opt/cray/pe/hdf5-parallel/1.12.1.5/crayclang/14.0/include -I /opt/cray/pe/pmi/6.1.3/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -I /opt/cray/xpmem/2.4.4-2.3_11.2__gff0e1d9.shasta/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -std=c++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build/Src -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=a1db559448e604e2 -fcuda-allow-variadic-functions -munsafe-fp-atomics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/AMReX_MLNodeLaplacian_sync-f6a335/AMReX_MLNodeLaplacian_sync-gfx90a.o -x hip /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp 1. parser at end of file 2. Code generation 3. Running pass 'CallGraph Pass Manager' on module '/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp'. 4. Running pass 'SI Optimize VGPR LiveRange' on function '@_ZN5amrex13launch_globalILi256EZNS_11ParallelForILi256EZNKS_15MLNodeLaplacian6refluxEiRNS_8MultiFabERKS3_S6_S4_S4_S6_EUliiiE3_EENSt9enable_ifIXsr19MaybeDeviceRunnableIT0_EE5valueEvE4typeERKNS_3Gpu10KernelInfoERKNS_3BoxEOS9_EUlvE_EEvS9_' #0 0x00000000038258cf PrintStackTraceSignalHandler(void*) Signals.cpp:0:0 #1 0x00000000038229ed SignalHandler(int) Signals.cpp:0:0 #2 0x00007fbdc691a8bf (/lib64/libpthread.so.0+0x168bf) #3 0x00007fbdc55fecda raise (/lib64/libc.so.6+0x4acda) #4 0x00007fbdc5600374 abort (/lib64/libc.so.6+0x4c374) #5 0x00007fbdc55f6cd9 __assert_fail_base (/lib64/libc.so.6+0x42cd9) #6 0x00007fbdc55f6d61 __assert_fail (/lib64/libc.so.6+0x42d61) #7 0x000000000126ee3e (anonymous namespace)::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector&) const SIOptimizeVGPRLiveRange.cpp:0:0 #8 0x0000000001272e93 (anonymous namespace)::SIOptimizeVGPRLiveRange::runOnMachineFunction(llvm::MachineFunction&) SIOptimizeVGPRLiveRange.cpp:0:0 #9 0x00000000028bdacf llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x28bdacf) #10 0x0000000002dea470 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dea470) #11 0x00000000024c7319 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0 #12 0x0000000002dec0c6 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dec0c6) #13 0x0000000003badbea clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, clang::FrontendOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x3badbea) #14 0x0000000004a84d55 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4a84d55) #15 0x0000000005601088 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x5601088) #16 0x000000000436c7b8 clang::FrontendAction::Execute() (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x436c7b8) #17 0x00000000042ec625 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x42ec625) #18 0x0000000004433a8a clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4433a8a) #19 0x0000000000df2365 cc1_main(llvm::ArrayRef, char const*, void*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xdf2365) #20 0x0000000000defa6b ExecuteCC1Tool(llvm::SmallVectorImpl&) driver.cpp:0:0 #21 0x0000000000d2eb86 main (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xd2eb86) #22 0x00007fbdc55e92bc __libc_start_main (/lib64/libc.so.6+0x352bc) #23 0x0000000000ded0f9 _start /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/../sysdeps/x86_64/start.S:120:0 clang-14: error: unable to execute command: Aborted (core dumped) clang-14: error: clang frontend command failed due to signal (use -v to see invocation) Cray clang version 14.0.2 (ecfd9ef4dfd5696cd449133c0da0293d503c2f21) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/share/../bin clang-14: note: diagnostic msg: Error generating preprocessed source(s). gmake[2]: *** [_deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/build.make:1406: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o] Error 254 gmake[2]: *** Waiting for unfinished jobs.... clang-14: /home/jenkins/llvm/lib/Target/AMDGPU/SIOptimizeVGPRLiveRange.cpp:497: void {anonymous}::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector&) const: Assertion `UseMI->isPHI() && "Uses should be PHI in Endif block"' failed. PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -flocal-restrict -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name AMReX_MLNodeLaplacian.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -resource-dir /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6 -dependency-file CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o.d -MT _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include/cuda_wrappers -idirafter /opt/rocm-5.2.0/include -include __clang_hip_runtime_wrapper.h -isystem /opt/rocm-5.2.0/hip/include -isystem /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -isystem /opt/rocm-5.2.0/include/hiprand -isystem /opt/rocm-5.2.0/include/rocrand -isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -isystem /opt/cray/pe/cce/14.0.2/cce/x86_64/include/craylibs -D __CRAY_X86_TRENTO -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__=1 -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build -I /opt/rocm-5.2.0/include -D NDEBUG -I /opt/cray/pe/mpich/8.1.18/ofi/cray/10.0/include -I /opt/cray/pe/libsci/22.08.1.1/CRAY/9.0/x86_64/include -I /opt/cray/pe/hdf5-parallel/1.12.1.5/crayclang/14.0/include -I /opt/cray/pe/pmi/6.1.3/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -I /opt/cray/xpmem/2.4.4-2.3_11.2__gff0e1d9.shasta/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -std=c++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build/Src -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=2f96d5e7b873b0d -fcuda-allow-variadic-functions -munsafe-fp-atomics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/AMReX_MLNodeLaplacian-3587ac/AMReX_MLNodeLaplacian-gfx90a.o -x hip /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp 1. parser at end of file 2. Code generation 3. Running pass 'CallGraph Pass Manager' on module '/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp'. 4. Running pass 'SI Optimize VGPR LiveRange' on function '@_ZN5amrex13launch_globalILi256EZNS_11ParallelForILi256EZNKS_15MLNodeLaplacian21restrictInteriorNodesEiRNS_8MultiFabES4_EUliiiE3_EENSt9enable_ifIXsr19MaybeDeviceRunnableIT0_EE5valueEvE4typeERKNS_3Gpu10KernelInfoERKNS_3BoxEOS7_EUlvE_EEvS7_' #0 0x00000000038258cf PrintStackTraceSignalHandler(void*) Signals.cpp:0:0 #1 0x00000000038229ed SignalHandler(int) Signals.cpp:0:0 #2 0x00007fba2efcd8bf (/lib64/libpthread.so.0+0x168bf) #3 0x00007fba2dcb1cda raise (/lib64/libc.so.6+0x4acda) #4 0x00007fba2dcb3374 abort (/lib64/libc.so.6+0x4c374) #5 0x00007fba2dca9cd9 __assert_fail_base (/lib64/libc.so.6+0x42cd9) #6 0x00007fba2dca9d61 __assert_fail (/lib64/libc.so.6+0x42d61) #7 0x000000000126ee3e (anonymous namespace)::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector&) const SIOptimizeVGPRLiveRange.cpp:0:0 #8 0x0000000001272e93 (anonymous namespace)::SIOptimizeVGPRLiveRange::runOnMachineFunction(llvm::MachineFunction&) SIOptimizeVGPRLiveRange.cpp:0:0 #9 0x00000000028bdacf llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x28bdacf) #10 0x0000000002dea470 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dea470) #11 0x00000000024c7319 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0 #12 0x0000000002dec0c6 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dec0c6) #13 0x0000000003badbea clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, clang::FrontendOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x3badbea) #14 0x0000000004a84d55 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4a84d55) #15 0x0000000005601088 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x5601088) #16 0x000000000436c7b8 clang::FrontendAction::Execute() (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x436c7b8) #17 0x00000000042ec625 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x42ec625) #18 0x0000000004433a8a clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4433a8a) #19 0x0000000000df2365 cc1_main(llvm::ArrayRef, char const*, void*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xdf2365) #20 0x0000000000defa6b ExecuteCC1Tool(llvm::SmallVectorImpl&) driver.cpp:0:0 #21 0x0000000000d2eb86 main (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xd2eb86) #22 0x00007fba2dc9c2bc __libc_start_main (/lib64/libc.so.6+0x352bc) #23 0x0000000000ded0f9 _start /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/../sysdeps/x86_64/start.S:120:0 clang-14: error: unable to execute command: Aborted clang-14: error: clang frontend command failed due to signal (use -v to see invocation) Cray clang version 14.0.2 (ecfd9ef4dfd5696cd449133c0da0293d503c2f21) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/share/../bin clang-14: note: diagnostic msg: Error generating preprocessed source(s). gmake[2]: *** [_deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/build.make:1392: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o] Error 254 gmake[1]: *** [CMakeFiles/Makefile2:1978: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/all] Error 2 gmake: *** [Makefile:146: all] Error 2 [mgarten@login2.crusher impactx]$ ```

Edit 2023/02/08: removed -DImpactX_PYTHON=ON in description here to make it simpler; add download instructions

n01r commented 1 year ago

Thanks to module changes suggested by Marcus Wagner (HPE), the HIP-version of ImpactX now also builds successfully! I will continue with simple example runs and try to profile them.

Replaced 

module load craype-accel-amd-gfx90a
module load rocm/5.2.0
module load cray-mpich
module load cce/14.0.2  # must be loaded after rocm

with

module load craype-accel-amd-gfx90a
module load rocm/5.3.0
module load cray-mpich/8.1.23
module load cce/15.0.0  # must be loaded after rocm

and added

module unload xalt/1.3.0