search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 23 forks source link

add option to nightqa to reprocess everything except darks #2401

Open akremin opened 4 weeks ago

akremin commented 4 weeks ago

Recently, nightqa has become much more variable in run time, likely correlated with the recent Perlmutter instabilities. For reference, a recent attempt to run nightqa on 20241015 took over 30 minutes before I killed the script.

Rerunning by hand shows that it is hanging when running preproc. Since we often run nightqa after manipulating only science data, it would be nice to have a simple option to tell it to reproduce all files except the darks, which wouldn't change in such circumstances and would avoid the issue of preproc occasionally hanging.

We should also investigate why nightqa hangs and improve that, but the above proposal could be implemented much faster.

sbailey commented 4 weeks ago

nightqa could also use desispec.util.runcmd(desispec.scripts.preproc.main, args, inputs=..., outputs=...) so that if the dark preproc outputs already exist, it wouldn't re-generate them.

akremin commented 4 weeks ago

Thanks, Stephen, that was my initial thought too but that doesn't work as implemented for two reasons. First, nightqa saves the preproc outputs to a temporary directory that is unique to each instance, so rerunning nightqa won't find the old preproc files. Second, this request is specifically for the situation where we want to redo all of the nightqa outputs, so it removes all of them by default. That doesn't mean it has to remove the preproc files, but if you're removing and redoing the dark QA you may expect it to also rerun preproc on the morning dark, since that isn't handled by the normal pipeline and data may have been flagged or altered.