We used to run pbmpi on 30 nodes with 16 tasks per node. We had no issues running it for months when it was compiled with GCC and OpenMPI.
Recently we compiled it with GCC+MPICH and Intel+Intel-MPI and none of those versions are running. The program hangs. I could progress a bit by adding some synchronisations (barriers + synchronous sends) but still, the program would hang after a few iterations.
We used to run pbmpi on 30 nodes with 16 tasks per node. We had no issues running it for months when it was compiled with GCC and OpenMPI.
Recently we compiled it with GCC+MPICH and Intel+Intel-MPI and none of those versions are running. The program hangs. I could progress a bit by adding some synchronisations (barriers + synchronous sends) but still, the program would hang after a few iterations.