healpix redshifts only use good tiles

[sbailey]

healpix redshifts should only use data from good tiles, not all tiles processed. This requires checking svn $DESI_SURVEYOPS/ops/tiles-specstatus.ecsv for QA=='good' and ZDONE=='true' (i.e. both QA approved and archived by the daily ops).

This update would allow us to process partially completed tiles and tiles that haven't been QA approved, without having their spectra end up in the healpix redshifts. Previously we thought that any exposures that weren't good enough for healpix redshifts would just be flagged as bad, but we hadn't considered the boundary cases with incomplete tiles — we don't want to flag them as permanently bad, but we also don't want them to be included in final healpix redshifts either.

[akremin]

An alternative to this (discussed offline with Stephen) is to have this logic in the desi_run_night script triggered by a flag such that exposures from unfinished tiles are not processed. This has the benefit that it prevents tiles-based studies from ingesting and using unfinished spectra. The downside, however, is that it provides fewer spectra in the release and some unfinished tile spectra will have enough S/N to be viable for some analyses.

[sbailey]

@akremin thanks for reminding me of that conversation. Indeed, I prefer that desi_run_night pre-filtering as the default so that standard productions only include QA-approved good tiles, avoiding the confusion of having cframes / tiles present that aren't in the healpix files.

[akremin]

This is implicitly solved by having desi_run_night only process "good" tiles by default for productions. However, if we wanted belt-and-suspenders, we could have the redshift submission script also check for "good" tiles. I purposely factored the code out into a function desispec.scripts.submit_night.get_completed_tiles https://github.com/desihub/desispec/blob/234e5b6890927f170503df72fbc2d3a2e82d63e8/py/desispec/scripts/submit_night.py#L466 so that it could be reused. It takes an optional path argument but will look in the svn location at NERSC by default, and returns a numpy array of tileid's that pass the selection.