Does amd GPU on multi-node still be tested？

[jsboer]

Hi， the document says "multi-GPU runs on mutliple nodes have not been tested", does elpa is still the situation now? We want to use elpa on an AMD GPU cluster, can elpa do it now?

[marekandreas]

Hi, yes ELPA does run on multi GPU, multi nodes setups for NVIDIA, AMD, and Intel GPUs. Where did you still find the "multi-GPU runs on mutliple nodes have not been tested" statement?

[jsboer]

Oh, thank you! It is in the https://github.com/marekandreas/elpa/tree/master/documentation, specially in https://github.com/marekandreas/elpa/blob/master/documentation/INSTALL.md#22-enabling-gpu-support.

AMD GPUs:

multi-GPU runs on one node have been tested (with MPI)
multi-GPU runs on mutliple nodes have not been tested

[jsboer]

Hi, yes ELPA does run on multi GPU, multi nodes setups for NVIDIA, AMD, and Intel GPUs. Where did you still find the "multi-GPU runs on mutliple nodes have not been tested" statement?

Hi, since ELPA support run on multi GPU, multi nodes setups for AMD GPUs, how to install ELPA with AMD GPU and MPI? I notice when use NVIDIA GPU with MPI, one can setup mpi compiler for cpp and fortran by CC and FC respectively, while GPU compiler is passed by --with-cuda-path. But there is no such a --with-amd-path shown in configure.

[marekandreas]

Typically a configure line like this works on AMD systems: ../configure CPP="gcc -E" CC=cc CXX=hipcc FC=ftn CXXFLAGS="-DROCBLAS_V3 -D__HIP_PLATFORM_AMD__ --offload-arch=gfx90a -g -O3 -std=c++17" CFLAGS="-g -O3 -std=c++17" LIBS="-lamdhip64 -fPIC" FCFLAGS="-g -O3" --enable-option-checking=fatal --with-mpi=yes --disable-sse --disable-sse-assembly --disable-avx --disable-avx2 --disable-avx512 --enable-amd-gpu --enable-single-precision --enable-gpu-streams=amd --enable-hipcub --disable-cpp-tests --with-rocsolver