Open zack-hable opened 5 years ago
Sorry for the very late response, if it is still applicable, I have a few questions about this issue:
V1.10.7 : This works as expected (I cannot use this version as it will not support EFA networking)
you require EFA networking, is this an issue that you experience with EFA instance types or just with t2.micro and TCP?
Background information
What version of Open MPI are you using?
Describe how Open MPI was installed
wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz
tar xzvf openmpi-4.0.1.tar.gz
cd openmpi-4.0.1
./configure --prefix=/shared/mpi --with-libfabric=/opt/amazon/efa/ --with-sge
sudo make install all
Please describe the system on which you are running
Details of the problem
Steps to reproduce
LD_LIBRARY_PATH=/shared/mpi/lib:/opt/amazon/efa/lib64 /shared/mpi/bin/mpic++ client.cpp -o client
LD_LIBRARY_PATH=/shared/mpi/lib:/opt/amazon/efa/lib64 /shared/mpi/bin/mpic++ server.cpp -o server
LD_LIBRARY_PATH=/shared/mpi/lib:/opt/amazon/efa/lib64 /shared/mpi/bin/ompi-server --no-daemonize -r + -d &
qsub Server.job
qstat
qsub Client.job
Observed Behavior
When running a server job via SGE (or slurm and torque) that spans multiple machines and attempting to connect with a client results in the client getting a seg fault in the mpirun executable. However when the job only runs on a single machine then the connection works. (use
-pe mpi 1
in Server/Client.job files)Expected Behavior
The clients should connect to each server instance and then send "Hello World" to one of them. (ie: client 1 and 2 connect to server 1 and 2, but client 1 sends to server 1 and client 2 sends to server 2)
Notes
Other Versions
I've tested the other following versions of OpenMPI (same install procedure as above) and the observed output was:
Output
client.out
server.out
Source Files
client.cpp
server.cpp
Client.job
Server.job