automate tile grouping, coadds, and redshift fitting

[sbailey]

The standard pipeline automates grouping spectra by healpix and running redshifts, but for (mini)SV we're running desi_group_tileframes to group by tiles/nights, and then desi_coadd_spectra and running rrdesi on that. Automate that process.

What I ran for 20200219 by hand:

#- Group by tile/night
cd /global/cfs/cdirs/desi/spectro/redux/daily
desi_group_tileframes -i exposures/20200219 -o tiles

#- Generate coadds per-spectrograph per-tile per-night
export NIGHT=20200219
for TILEID in 63075 70003 70004 70005; do
    cd /global/cfs/cdirs/desi/spectro/redux/daily/tiles/$TILEID/$NIGHT
    for SPECTRO in 0 3 6 7 9; do
        coadd=coadd-${SPECTRO}-${TILEID}-${NIGHT}.fits
        if [ ! -f $coadd ]; then
            desi_coadd_spectra --coadd-cameras --nproc 8 \
                -i cframe-?${SPECTRO}-*.fits \
                -o coadd-${SPECTRO}-${TILEID}-${NIGHT}.fits
        else
            echo $coadd exists
        fi
    done
done

#- On a 4 node interactive job, ~1.5 min per coadd:
salloc -N 4 -C haswell -t 04:00:00 -q interactive
export NIGHT=20200219
for TILEID in 63075 70003 70004 70005; do
    cd /global/cfs/cdirs/desi/spectro/redux/daily/tiles/$TILEID/$NIGHT
    for SPECTRO in 0 3 6 7 9; do
        zbest=zbest-${SPECTRO}-${TILEID}-${NIGHT}.fits
        if [ ! -f $zbest ]; then
            echo running redrock for $zbest
            time srun -N 4 -n 128 -c 2 rrdesi_mpi \
                -z $zbest \
                -o redrock-${SPECTRO}-${TILEID}-${NIGHT}.h5 \
                coadd-${SPECTRO}-*.fits
        else
            echo $zbest exists
        fi
    done
done

Using a single node (instead of 4) takes ~3 min per coadd, and these could be spawned as N independent single-node jobs per night in the realtime queue. The previous steps of grouping and coadding per night are fast enough they could be done on the interactive node prior to submitting the redshift jobs.

[sbailey]

Update: The script to do this was added in #892. We're still getting enough hiccups (mostly with arc processing) that I don't want to turn this on as a daily cronjob or otherwise automatically launched job just yet.

[sbailey]

I'm leaving this open until we've fully automated it in the daily processing, but for the purposes of the minisv2 project I've flagged this to the "Done" column and we'll just run this by hand.

[sbailey]

Update: the script in #892 runs out of time on nights with lots of different tiles (e.g. nights with dither tests). Update the wrapper script to create all the cframe links, and then submit one job per tile to do the coadds and run redshifts.

Also: add the creation of spectra files per-tile per-night and run rrdesi on that instead of the coadd (though still create the coadd).

Also: switch the coadd to being across exposures but not across cameras (i.e. drop the --coadd-cameras option).