standardize on workflow scripts

AMReX-Astro / Castro

Castro (Compressible Astrophysics): An adaptive mesh, astrophysical compressible (radiation-, magneto-) hydrodynamics simulation code for massively parallel CPU and GPU architectures.

http://amrex-astro.github.io/Castro

Other

300 stars 98 forks source link

standardize on workflow scripts #73

Closed zingale closed 5 years ago

zingale commented 7 years ago

We all have our own job workflow scripts. We should take the best (and simplest/cleanest) of them and come up with a standard set of job workflow processes.

Things they should address include:

submission script with best practices for each machine / code
automagic detection of checkpoint files and automatic restart in the same script
automatic archiving to HPSS as output files are created
post processing hooks
slack integration (e.g., post in a channel when a run starts)

The job submission script itself needs to be a shell script (bash seems to work fine)

The remaining tools could probably be python

zingale commented 7 years ago

@guadabsb15 @maxpkatz @adam-m-jcbs @dwillcox

zingale commented 7 years ago

my current scripts for edison in Castro/Util/job_scripts/edison/:

edison.MPI.slurm : this will find the checkpoint file for restart automatically
edison.xfer.slurm , process.xrb : this will run on the transfer nodes and will look for output and automatically archive to HPSS.

zingale commented 5 years ago

this is now part of the separate workflow repo