Sky subtraction failure exp 74464 for tile 80699 due to LED light leak

[djschlegel]

Ting-Wen Lan identified some failed truth-table redshifts that appeared to be due to z-band sky-subtraction failure, shown on slide 12 of his presentation posted to https://desi.lbl.gov/trac/wiki/DataSystems/Telecons/2021-07-27#no1 . This appears to trace back to an obvious failure of sky-subtraction on a single exposure+ccd, exp 74464 camera z0 on night 20210203.

The raw ccd frame should be on the list of rejected frames, since this one has the problem of the LED light leak through the shutter. Attached is a screen shot of the bottom of the raw image, and then the corresponding sky-subtracted file sframe-z0-00074464.fits .

[sbailey]

@djschlegel thanks. In Everest I see this was already flagged with BADCAMWORD=a3z0 (i.e. b3,r3,z3,z0 should have been unprocessed) but somehow z0 make it through cframes and into redshifts anyway.

I'll manually remove this from everest, but we should scrape logs to figure out why the BADCAMWORD didn't stop the z0 processing so that we don't trip on this for Fuji.

[sbailey]

Status update: due to stdstar pipeline logic problem in #1225, I had to remove all of sp0 not just z0 for expid 74464. After that I regenerated the stdstar fit for expid 74465 (same tile same night, but the previous cross-exposure stdstar fit had included the bad z0) and then redid fluxcalib, cframe, and impacted spectra/coadds/redshifts.

For the purposes of Everest I will call this done, but leave it open to double check for Fuji that it doesn't accidentally get reprocessed. I think this may have been retroactively flagged as bad after the initial Everest pipeline run but I'm not completely sure about that, and this is a good test case to check #1225 fixes too.

[akremin]

As guessed by Stephen, this cframe was produced before the zcam was flagged in the exposure table. Now that this and others are properly flagged, they will not be an issue for Fuji or any future production. However, I did identify other cases where this occurred. There are four other examples similar to the one flagged here and nine where the cframes don't exist but the cameras were used for standard star fitting and therefore impacted good cameras. Those lists are below.

I will also add that due to the #1225 bug, these petals/spectrographs would have failed in the automated pipeline (and indeed on similar tiles they did). So we can also consider just removing the affected files without rerunning to re-generate using the remaining exposures, since the automated pipeline wouldn't have generated those either. Though leaving redshifts behind is always painful, so I applaud the extra-effort to get what we can.

Additional exposures that had a saturated zcam and produced a bad cframe: night,expid,camword 20210221,77430,z2 20210318,80961,z8 20210319,81101,z2 20210322,81533,z0

Exposures where standard stars were fit using the bad zcam but the cframe doesn't exist (less bad, but the standard star fits will be sub-optimal and should be rerun): night,expid,camword 20210306,79574,z7 20210309,79877,z8 20210309,79885,z28 20210327,82505,z2 20210328,82608,z7 20210406,83737,z2 20210411,84350,z2 20210412,84517,z6 20210412,84535,z6

Note 1: Night 20210309, exp 79885, had two z cameras: z28 Note 2: Both nights 20210309 and 20210412 have two different exposures (but on different tiles)

[sbailey]

Thanks for the augmented list @akremin. I have purged those spectrograph exposures from everest and rerun stdstars/cframes/coadds/redshifts for files that were impacted via joint stdstar fits.

[astro-blue]

Hi all, I just run the comparison between Denali and the new Everest and show the redshift distributions for those fibers (fiberid 0-500) with inconsistent redshifts. The top panel shows the results I reported half month ago and the bottom panel shows the results with the new Everest catalog. The redshift distribution now makes sense.