Drop negative TARGETID and SKY* targets from healpix-based coadds

[sbailey]

Currently the healpix-based coadds propagate all TARGETIDs, including the negative TARGETIDs that arise from stuck positioners not being able to point at a normal DESI target. These are ~20% of the targets in the healpix files and are nearly useless. I suggest that we drop them from the healpix-based spectra, coadds, and redrock files.

I also suggest that we drop SKY, SUPP_SKY, and BAD_SKY targets from the healpix groupings. Custom studies that want to use sky fibers could/should use tile-based data instead.

Any objections?

[moustakas]

I agree that this is a sensible idea.

On Tue, Oct 5, 2021, 8:06 PM Stephen Bailey @.***> wrote:

Currently the healpix-based coadds propagate all TARGETIDs, including the negative TARGETIDs that arise from stuck positioners not being able to point at a normal DESI target. These are ~20% of the targets in the healpix files and are nearly useless. I suggest that we drop them from the healpix-based spectra, coadds, and redrock files.

I also suggest that we drop SKY, SUPP_SKY, and BAD_SKY targets from the healpix groupings. Custom studies that want to use sky fibers could/should use tile-based data instead.

Any objections?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/desihub/desispec/issues/1433, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK5SDHPSGJ47TQGGKY2323UFOHIDANCNFSM5FNAMR3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[djschlegel]

Has anyone checked with the LyA group. Most relevant would be if LyAF systematics expect to use these (though I think they'd be identical to the tile-based sky spectra, since these skies don't repeat on different tiles).

[stephjuneau]

In my opinion, it doesn't sound robust to have to use both healpix- and tile-based coadds with the knowledge that in the long term we might end up with only the healpix-based coadds. I think it would be useful to keep at least the small percentage of sky fibers that end up having a serendipitous object if they pass through RR and are found to have a valid spectrum/redshift in the zpix-* files. Lastly, I wonder whether they could be used to compute approximate sky spectra until we have proper sky models on a per-coadded-spectrum basis? We just had a discussion about the importance of knowing the sky when stacking DESI spectra (with @moustakas @biprateep, @weaverba137 and others) this morning, and this seems relevant.

[sbailey]

Point of clarification: these are sky-subtracted sky-spectra and sky-subtracted whatever-the-stuck-fiber-was-pointing-at, not directly sky spectra. i.e. it's useful for studying systematics of sky subtraction residuals, but not useful for studying the actual sky spectra that were subtracted.

Stuck positioners have a unique TARGETID per tile, so the zpix spectra don't contain any information that isn't already identical in the per-tile cumulative grouping. Since LSS uses the cumulative per-tile coadds+redshifts, we plan to retain those in future prods in addition to the healpix ones (perexp and pernight may get dropped, but not cumulative).

Pre-identified sky fibers might be re-used on other tiles and thus could be differently coadded in healpix tiles, but they were specifically selected because nothing was there in the imaging data, so any serendipitous spectra would be a transient where the shorter timescale per-tile spectra would be more useful than a healpix coadd anyway.

We do need to followup with LyA about this since there might be a big convenience factor to using healpix for everything not not mixing and matching directory structures for sky-fiber systematics studies. @alxogm @andreufont et al? Is this worth 20% extra disk space?

[andreufont]

I would imagine that careful studies of systematics would always use the per-tile spectra, but the real expert here is @julienguy.

[sbailey]

I'm not convinced that we'll really use these healpix-based sky spectra, but erring on the side of keeping data products, we'll keep these for Fuji. If they never get used we may drop them for DA1 (Kilimanjaro?)