search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

enable specex MPI parallelism from command line #748

Closed sbailey closed 3 years ago

sbailey commented 5 years ago

desispec.scripts.specex.main has options for MPI parallelism that are used by the pipeline, but these are not exposed by the desispec/bin/desi_compute_psf wrapper script. Add an --mpi or equivalent option to desispec/bin/desi_compute_psf so that it can be called standalone with MPI parallelism independently of running a production.

tskisner commented 5 years ago

A note on this. Currently the pipeline logs, for each task, the equivalent command-line to reproduce how it ran that task. However, there are 2 steps (PSF estimation and extraction) with are run with MPI by the pipeline but which do not have MPI-enabled entry points for manual use. So, this issue also includes the extraction entrypoint script.

marcelo-alvarez commented 3 years ago

This has been implemented in desispec/bin/desi_compute_psf_mpi, which is the equivalent of adding an --mpi option to desispec/bin/desi_compute_psf, so there exists a wrapper script capable of calling specex standalone with MPI parallelism independently of running a production. Closing this issue.