Mismatch of sending and receiving of status in PFASST

[torbjoernk]

The really weird mismatch of the blocking status communication is still there on the feature/fix-mpi-branch of PR #212.

I put in some debug logging of the following in mpi_communicator_impl.hpp:

CLOG(DEBUG, "Controller") << "sending converged status to rank " << dest_rank << " with tag '1': " << ((bool)iconverged == IStatus::CONVERGED);
int err = MPI_Send(&iconverged, sizeof(int), MPI_INT, dest_rank, 1, mpi->comm);
check_mpi_error(err);
CLOG(DEBUG, "Controller") << "sent converged status";

and

CLOG(DEBUG, "Controller") << "recieving converged status from rank " << src_rank << " with tag '1'";
int err = MPI_Recv(&iconverged, sizeof(iconverged), MPI_INT, src_rank, 1, mpi->comm, &stat);
check_mpi_error(err);
CLOG(DEBUG, "Controller") << "recieved converged status from rank " << src_rank << " with tag '1': " << ((bool)iconverged == IStatus::CONVERGED);

and a grep on the logfiles of mpiexec -np 8 examples/advection_diffusion/mpi_pfasst --spatial_dofs 4096 --dt 0.01 --tend 2.56 --num_iter 30 --abs_res_tol 1e-10 -q yields:

$> grep -c 'sending converged status' mpi-rank-000*.log
mpi-rank-0000.log:93
mpi-rank-0001.log:100
mpi-rank-0002.log:107
mpi-rank-0003.log:158
mpi-rank-0004.log:168
mpi-rank-0005.log:211
mpi-rank-0006.log:282
mpi-rank-0007.log:0
$> grep -c 'sent converged status' mpi-rank-000*.log
mpi-rank-0000.log:93
mpi-rank-0001.log:100
mpi-rank-0002.log:107
mpi-rank-0003.log:158
mpi-rank-0004.log:168
mpi-rank-0005.log:211
mpi-rank-0006.log:282
mpi-rank-0007.log:0
$> grep -c 'recieving converged status' mpi-rank-000*.log
mpi-rank-0000.log:0
mpi-rank-0001.log:93
mpi-rank-0002.log:97
mpi-rank-0003.log:95
mpi-rank-0004.log:158
mpi-rank-0005.log:168
mpi-rank-0006.log:211
mpi-rank-0007.log:247
$> grep -c 'recieved converged status' mpi-rank-000*.log
mpi-rank-0000.log:0
mpi-rank-0001.log:93
mpi-rank-0002.log:97
mpi-rank-0003.log:95
mpi-rank-0004.log:158
mpi-rank-0005.log:168
mpi-rank-0006.log:211
mpi-rank-0007.log:247

[torbjoernk]

Fixed by @memmett in PR torbjoernk/PFASST#5