Closed egrace479 closed 4 months ago
There are still some questions to be answered on overall package and things to do after this update is settled:
mpi_downloader
to distributed_downloader
in src
. We then have to decide how to work with the other scripts outside of it but still in src
. The idea is that src/submitter.py
will be the interface that runs the whole download, while the scripts/
and config/
files will be modified by users to pass into the process.tools
folder or similar at the root level of the repository. I'm fine with leaving them not generalized for now, but good to include the tools.server_prep
and MPI_download_prep
--what goes in and what actually comes out).
The sample
.env
file now defines paths previously defined in the submitted script. It also has the project name, so that and all the distribution of work settings can be passed to theslurm
scripts directly through the bash coordination script (example_scripts/submit_mpi_download.sh
) instead of needing to be redefined at the top of each file. This does add a slightly higher resource request for the verifier, but (following our group discussion) this shouldn't add much to resource allocation since it is only run for a couple seconds on each batch.src/submitter.py
is now the primary user interface with the package (following pre-processing steps). Its only input is the path to the.env
file, so that it can fetch the required paths. There was also some restructuring to avoid redefining variables.