zcatalog should include SURVEY / FAPRGRM / ZGROUP / NIGHT

[sbailey]

Combined zcatalogs should include header keywords to record the SURVEY, FAPRGRM, ZGROUP that went into them for the cases then they are all consistent, e.g.

e.g. zcatalog/ztile-main-dark-pernight.fits should have SURVEY=main, FAPRGRM=dark, ZGROUP=pernight so that information isn't available only in the filename.

When implementing, consider special cases of combining arbitrary input catalogs so that we don't accidentally break the ability to combined multiple different inputs into one.

The same concepts and caveats likely apply to the input redrock catalogs too.

[sbailey]

Related: ztile-*-pernight.fits files should include a NIGHT column; currently for repeat TARGETID, TILEID entries we lack the information for which night goes with which row.

[sbailey]

From #1394 : for the healpix-based combined redshift catalogs (or even the non-healpix ones), it would be convenient to include the HPXPIXEL that goes with a target to facilitate tracking an entry back to the original spectra. This is derivable from TARGET_RA, TARGET_DEC by knowing NSIDE=64 and NESTED=True, but that also could be error prone from non-expert end-users.

Related: desimodel.footprint.radec2pix(nside, ra, dec) takes care of degrees vs. radians, polar coordinates vs. ra,dec, nested=True, etc. so that the only special knowledge is that the spectro pipeline uses nside=64.

[AlexGKim]

@sbailey . I have two quasi-real time active pipelines that expect NIGHT in the table. Is it acceptable to have an add-on that takes output redrock*.fits and overwrites them with an added NIGHT column. And have this run up to Fuji?

[sbailey]

@alex please post a specific file where you are tripping on a missing NIGHT column. For a quasi-realtime pipeline I expect that you are using the daily pipeline, but that is only running "cumulative" redshifts that can potentially coadd data across nights and thus "NIGHT" doesn't apply as a column in the FIBERMAP, though the new EXP_FIBERMAP HDU retains the information about the individual exposures that went into the coadd, and thus includes NIGHT.

pernight coadds in a production could arguably have an extra NIGHT column, at the expense of yet another special case for what columns are in a FIBERMAP (across tiles, single tile across nights, single tile within a night, uncoadded...). I'll also email desi-data with more details.

[stephjuneau]

@sbailey are you tracking NEXP, NNIGHT and NTILE for the number of exposures, nights, and tiles that went into any coadd (analogous to NOBS_G, R, Z tracking how many observations went into each band in the photometry)? If not, would this be useful to help understand the coadd files and know what to expect when searching/joining with files/HDUs from individual exposures, nights or tiles?

[sbailey]

@stephjuneau coadded FIBERMAPs have columns COADD_NUMEXP, COADD_EXPTIME, COADD_NUMNIGHT, and COADD_NUMTILE, with the caveat that due to a bug reported in #1349, they are recording the number of things that could have gone into the coadd, not the number that actually did go into the coadd after quality selections may have discarded bad data before coadding.

[AlexGKim]

@sbailey , thanks for your patience and help. Indeed the EXP_FIBERMAP HDU meets my needs.

[sbailey]

Implemented in PR #1618, though SURVEY and FAPRGRM require the use of desi_zcatalog --header option to set them, since we don't reliably have those keywords in upstream files to propagate.