Posting this as something that will become an issue in the future - the current CI environments use a version of OpenMPI that is compatible with the test suite.
Describe the bug
The set of tests that compares the output of MPI/multiprocess/single process workflows invokes mpirun as a subprocess. Because PyRate imports and initialises MPI itself before the subprocess is started, this counts as a "recursive" invocation of MPI, which is not supported by OpenMPI >=3.0.0. This means that these tests will fail with a CalledProcessError on systems with newer versions of OpenMPI (and worse, will not give useful debug information - just an exit code 1).
Install OpenMPI, GDAL, proj and pyrate develop branch on Ubuntu 20.04 (the default OpenMPI version on 20.04 is 4.1.1), and install the pyrate test dependencies.
Try to run pytest tests/test_mpi_vs_multiprocess_vs_single_process.py from the root of the repository.
Expected behavior
The MPI tests should work with newer versions of OpenMPI, or at least automatically skip if a version incompatible with the test suite is detected.
Desktop (please complete the following information):
OS: issue is present on Ubuntu 20.04, MacOS 11.5.2 with homebrew and CentOS Stream 8 (NCI OOD/VDI)
Version: tested with the develop branch of PyRate.
Additional context
This will impact the ability of users to run the test suite and confirm that PyRate works with MPI on their systems. For now, it doesn't affect the CI environments because these use Ubuntu 18.04, and the default OpenMPI version for that OS is 2.1.1 which doesn't exhibit this issue. A workaround to run the tests on a newer OS is to either manually install the older version of OpenMPI or use MPICH instead.
Posting this as something that will become an issue in the future - the current CI environments use a version of OpenMPI that is compatible with the test suite. Describe the bug The set of tests that compares the output of MPI/multiprocess/single process workflows invokes
mpirunas a subprocess. Because PyRate imports and initialises MPI itself before the subprocess is started, this counts as a "recursive" invocation of MPI, which is not supported by OpenMPI >=3.0.0. This means that these tests will fail with aCalledProcessErroron systems with newer versions of OpenMPI (and worse, will not give useful debug information - just anexit code 1).See this mpi4py issue for more details.
To Reproduce Steps to reproduce the behavior:
pytest tests/test_mpi_vs_multiprocess_vs_single_process.pyfrom the root of the repository.Expected behavior The MPI tests should work with newer versions of OpenMPI, or at least automatically skip if a version incompatible with the test suite is detected.
Desktop (please complete the following information):
Additional context This will impact the ability of users to run the test suite and confirm that PyRate works with MPI on their systems. For now, it doesn't affect the CI environments because these use Ubuntu 18.04, and the default OpenMPI version for that OS is 2.1.1 which doesn't exhibit this issue. A workaround to run the tests on a newer OS is to either manually install the older version of OpenMPI or use MPICH instead.