Closed sbailey closed 1 week ago
I have a very basic shell script that performs some of this functionality. I can provide details tomorrow.
This script handles the basic logic of:
The script can be wrapped in a batch job submitted to the workflow queue.
Implemented in PR #2322 ; closing ticket.
One of the hassles of running a production is that it requires more jobs than the NERSC 5000 limit in the regular queue. Currently this is handled by humans monitoring the queue and launching additional nights/months when it drops below some threshold of pending jobs. This could be automated with something like an hourly scronjob script to
Ideally this is a wrapper to existing job launching infrastructure and doesn't require any additional changes to the processing tables or other bookkeeping. Items to work out:
desi_proc_night -n NIGHT ...
, e.g. due to a transient sbatch hiccup. If it got far enough to have a partial processing table written, then the above algorithm would think the night was already submitted and semi-incorrectly proceed with the next night when it ran again. But trying to derive if the night is only partially submitted is potentially tricky, and potentially problematic if the launcher keeps trying and failing on a night.desi_resubmit_queue_failures -n NIGHT ...
)? Could be nice, but would need additional bookkeeping to limit the number of attempts. Feels like a beyond-the-baseline feature that could risk getting the original feature done in time for Kibo.