desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
33 stars 24 forks source link

tweaks for running jura #2242

Closed sbailey closed 1 month ago

sbailey commented 1 month ago

This PR includes 3 updates to make running Jura just a bit easier:

Details:

write_traces_in_psf tested with

desi_compute_trace_shifts -i /dvs_ro/cfs/cdirs/desi/spectro/redux/jura/preproc/20220104/00116769/preproc-b4-00116769.fits.gz --psf /dvs_ro/cfs/cdirs/desi/spectro/redux/jura/calibnight/20220104/psfnight-b4-20220104.fits --degxx 2 --degxy 0 --continuum --outpsf $SCRATCH/psf.fits

that also normally worked before, but checks that I don't have typos.

tilenight job runtimes from jura so far: image The orange line gives the current job runtime limit, which is why the dots don't exceed that line. I wanted to give a little more time while still keeping the nexp=1 case under the 30 minute debug queue limit (now 26 minutes).

queue_info_from_qids also tested with real-life usage parsing jura jobs, e.g. $CFS/desi/users/sjbailey/dev/jura/ccdcalib_runtime.py

Speaking of ccdcalib runtimes, I also considered increasing those job runtimes since they sometimes timeout. However, the current limit of 15 minutes is already pretty far into the tail of the regular distribution so I left it as is: image

@akremin after review, I suggest that we merge this, create a new incremental tag, and continue with Jura with this version.

sbailey commented 1 month ago

I also looked at flat job runtimes since those sometimes timeout. Our current limit of 20 minutes is already pretty far out on the tail so I think it is better to sometimes let them timeout rather than extend the limit further and waste even more compute time when a job hangs. image