Devel branch cause warning from OpenMPI

[bbsy789]

Compiler:GCC12.2.0 Mpi:Openmpi4.1.4 Petsc:3.18.0 System:Ubuntu

Running asfem binary program can see this warning.

--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              ln0
  Local adapter:           mlx5_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   ln0
  Local device: mlx5_0
--------------------------------------------------------------------------

This issues may be solved by this url https://github.com/open-mpi/ompi/issues/10841.

[yangbai90]

Dear @bbsy789,

I'm curious which input file have you tried? From my side, it works fine, I can't see this warning. I'm using gcc12.2.0+openmpi-4.1.4+petsc3.18.1 It could be a hardware dependent issue. I can try to execute the issue input file and figure out what occurs there.

[bbsy789]

Dear @yangbai90 , Actually, it's happening in starting asfem. my environment file following:

export PETSC_DIR=$HOME/AsFemLibs/petsc
export MPI_DIR=$HOME/AsFemLibs/openmpi
export BINUTILS_DIR=$HOME/software/binutils/2.39
export CLANG_DIR=/thfs1/home/liujinmei/software/toolchain/llvm
export PATH=$PATH:$MPI_DIR/bin:$GCC_DIR/bin:$HOME/AsFem-devel/bin:$CLANG_DIR/bin
export GCC_DIR=/thfs1/software/gcc/12.2.0
# please comment out the following four lines until the PETSc has installed !!!
export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export F90=mpif90

export OMP_NUM_THREADS=1

export C_INCLUDE_PATH=$MPI_DIR/include:$CLANG_DIR/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$MPI_DIR/include:$CPLUS_INCLUDE_PATH
export FPATH=$MPI_DIR/include:$CLANG_DIR/include:$FPATH
export MANPATH=$MPI_DIR/share/man:$CLANG_DIR/share/man:$MANPATH
export LD_LIBRARY_PATH=$HOME/AsFemLibs/aarch64-linux-gnu:$CLANG_DIR/lib:$GCC_DIR/lib:$GCC_DIR/lib64:MPI_DIR/lib:$PETSC_DIR/lib:$CLANG_DIR/lib:$LD_LIBRARY_PATH

And In PETSC_DIR folder, it's petsc 3.18.0. In MPI_DIR folder, it's openmpi 4.1.4. All of them is compiling by GCC 12.2.0. Compiling or installation process have not error. So I try to compile asfem again: git pull cmake -S . -B build -DCMAKE_CXX_COMPILER=$GCC_DIR/bin/g++ -DCMAKE_BUILD_TYPE=Debug -GNinja

-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Check for working C compiler: /thfs1/home/liujinmei/AsFemLibs/openmpi/bin/mpicc
-- Check for working C compiler: /thfs1/home/liujinmei/AsFemLibs/openmpi/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /thfs1/software/gcc/12.2.0/bin/g++
-- Check for working CXX compiler: /thfs1/software/gcc/12.2.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
We are running on linux system ...
MPI dir is: /thfs1/home/liujinmei/AsFemLibs/openmpi
PETSC dir is: /thfs1/home/liujinmei/AsFemLibs/petsc
AsFem will be compiled in Debug mode !
CXX compiler options are:  -Wall -Wextra -Werror -O2 -std=c++17
-- Configuring done
-- Generating done
-- Build files have been written to: /thfs1/home/liujinmei/AsFem-devel/build

cd build & ninja [189/189] Linking CXX executable ../bin/asfem

Then it's happening:

liujinmei@ln0:~/AsFem-devel$ asfem
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              ln0
  Local adapter:           mlx5_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   ln0
  Local device: mlx5_0
--------------------------------------------------------------------------
******************************************************************************
*** Welcome to use AsFem                                      AAA          ***
*** A Simple Finite Element Method Program                   // \\         ***
*** Version: 0.80        Release @ 2022-10-19               //   \\        ***
*** PETSc version:  3.18.0                                 //     \\       ***
*** License: GPL-3.0                                      //       \\      ***
*** Author: Yang Bai @ M3-Group                          //_________\\     ***
*** Contact: yangbai90@outlook.com                      //-----------\\    ***
*** QQ Group: 879908352                                //             \\   ***
*** Website: https://github.com/M3Group/AsFem         //               \\  ***
*** Feel free to use and discuss  .:.                **                 ** ***
******************************************************************************
******************************************************************************
*** Start to read the input file

[bbsy789]

I doubt the difference in the library version causes this problem. The first compete compiling start asfem failed due to missing a lot of libraries, such as libxxx.so, I use a mount of the library from this folder /usr/lib/aarch64-linux-gnu

[yangbai90]

It confuses me, so what will happen if you use:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -GNinja

where you can ignore the -DCMAKE_CXX_COMPILER= term, and let CXX and CC be mpicxx and mpicc compiled by your gcc12.