make the batch submisson cluster agnostic

HopkinsIDD / flepiMoP

The Flexible Epidemic Modeling Pipeline

https://flepimop.org

GNU General Public License v3.0

9 stars 4 forks source link

make the batch submisson cluster agnostic #191

Open jcblemai opened 7 months ago

jcblemai commented 7 months ago

There are hardcoded paths and slurm loaded modules, these need to be provided as separate scripts -- that the user can override from its home directory, such that we can run on any slurm cluster

TimothyWillard commented 2 months ago

@shauntruelove I saw you assigned this issue to me and it's labeled as high priority, do we have a desired outcome and deadline for this since this is now high priority? I'm not super familiar with the details of this issue, @jcblemai or @shauntruelove would you be able to fill those in?

shauntruelove commented 2 months ago

Notes:

need agnostic handling of secrets
remove hard-coded specifications with Rockfish
make work for Longleaf and others
add workflow section of the config to tell where everything goes

TimothyWillard commented 1 month ago

@pearsonca @jcblemai I've attached the branch I'm doing the documentation updates to to this issue.

jcblemai commented 1 month ago

[twillard@longleaf-login6 ~]$ sg rc_jlessler_pi -c "touch foobar.txt"
[twillard@longleaf-login6 ~]$ ls -lah foobar.txt
-rw-r--r-- 1 twillard rc_jlessler_pi 0 Sep 13 11:25 foobar.txt

We need to ensure that everyone can access the run, so commands should be optionally run for a configurable certain group of user. su requires sudo, sg does not (edited)

TimothyWillard commented 1 month ago

Update as of EOD 2024-09-16:

It is now possible to cleanly and consistently install flepiMoP on Longleaf using the build/hpc_install.sh script on the GH-191/longleaf-batch-submission branch. Yet to add rockfish, pending access to be able to work through adding that as a supported cluster.
Corresponding documentation changes have been started on the GH-191/longleaf-batch-submission-documentation branch
Added support for providing a slurm --partition to the batch/inference_job_launcher.py script, see bba583a.
~Still yet to successfully submit a batch run on Longleaf.~ Managed to successfully submit a batch run on Longleaf, yet to verify it worked correctly.