etneil
commented
7 years ago This is now handled by importing of the appropriate "local_taxi.py" file, which specifies the right MPI invocation as a string. The "localize" custom setuptools function picks out the appropriate local_taxi (and other local files) for a given machine, prior to installation of the package.
On Janus and Fermilab, mpirun can be called simply like "mpirun". However, on CU, a specific mpirun binary must be called by abspath. In general, it's a good design feature to be able to call non-default mpirun binaries.
Ideally, localization is limited to taxi.sh, local_taxi.py, and local_util.py. The runner scripts should be machine-agnostic. However, this raises the issue of how to gracefully tell the runner scripts how to call mpirun.
Currently, the way this is handled is: taxi.sh sets an environment variable "MPI_BINARY", which the runner scripts then evaluate to call mpirun. This is a terrible way to do this: security issues, and it means to use the runner scripts anywhere you must first "setenv MPIRUN='mpirun'" even if you don't require any special mpirun binary.
The correct solution is probably something like giving the runner scripts a command-line argument like "--mpibin" with the default "mpirun". However, any special mpirun binary more-or-less must be specified in taxi.sh. How do we get this information from taxi.sh, through taxi.py, to the runner scripts?
This scheme still makes the runner scripts more difficult to use on their own (not called by taxi) on systems where special mpirun binaries must be called (CU beowulf cluster). Every time you use a runner script on such a system, you must provide the special mpi binary. Is there some graceful way of making the runner scripts aware of a "localized default" without editing them directly?