r9 on 20211212 extractions are problematic in iron

[schlafly]

@akrolewski identified an issue on P9 on 20211212 where the LRG success rate was poor. This night looks fine in daily, but in @araichoor 's run of nightqa for iron, things are clearly bad. e.g., c.f. the daily and iron sky spectrum distributions here. https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20211212/skyzfiber-20211212.png https://data.desi.lbl.gov/desi/users/raichoor/spectro/iron/nightqa/20211212/skyzfiber-20211212.png The residuals in the sframes also look bad.

I don't know what the issue is, but it may be relevant that the psfnight-r*.fits are marked as red/missing for iron, and blue/symlinked for daily. So conceivably we fell back on default psf fits for 20211212, and those were good enough for most cameras but somehow exploded for r9??

[sbailey]

In addition to the tiles observed on night 20211212, tile 7733 was re-observed on 20211215 (exposure 114223) and thus included in the cumulative redshifts for 20211215. All other impacted tiles have LASTNIGHT=20211212.

[forero]

I began comparing sky sframes between daily and a given data release. I just subtracted the two sky sframes (for each camera and petal) calculated the average and the standard deviation of that difference. It's clear that those differences for 20211212 look quite different when comparing the daily data to iron, as seen when I compared to some date I chose (20210707 in daily vs. guadalupe).

[forero]

Here are more details about my process. On a given night, I read all the available sframes and specifically selected the sky fibers. For each exposure, petal, and camera, I calculated the difference between two sky sframes from different releases. This difference represents the iron sky data subtracted from the daily sky data for each respective exposure, petal, and camera.

Now, let's focus on two nights that seem to display typical behavior: 20210606 and 20220430. The analysis results for these nights consistently reveal mean distributions between -2 and 2 and standard deviations between 1 and 1000.

However, when I examine the problematic night, 20211212, I notice that the standard deviations exceed 1000 and the distribution of mean values appears notably broader on this night.

During my exploration, another night emerged as notably distinct from the rest - 20220613, which corresponds to the Iron cutoff date. On this night, both the mean value and standard deviation distributions differ significantly when compared to the first two nights I mentioned.

[schlafly]

Poking a little at 20230613, I think the most significant thing about it is just that it was all backup tiles, and the sframes for backup tiles look bad because they're so bright. Some suggestions:

• For the standard deviation, let's do something robust if you're not already.
• Something like mad_std(sframe_iron - sframe_daily), but instead mad_std((sframe_iron - sframe_daily)(ivar_iron ivar_daily)**(1/4)), might cale the residuals in a more sensible way.
• If the x axis were not the mean difference but instead the actual mean sky fiber spectrum, we'd be able to see which points had really bright skies.
• It might make sense to color the points by dark / bright / backup.

[akremin]

I have reprocessed the night in a special SPECPROD=reproc_20211212_iron. It can be found at NERSC here: /global/cfs/cdirs/desi/spectro/redux/reproc_20211212_iron

What I've done:

• Copied the bias and badcolumn calibration files from 20211212 in Iron.
• Copied the psfnight and fiberflatnight calibration files from 20211211 in Iron (because the arcs on the 12th have issues).
• Copied exposures 00112068, 00113429, 00113431, and 00114223 from nights 20211203, 20211211, 20211211, and 20211215 respectively from Iron.
• Ran all science jobs from 20211212 using the Iron tag.
• Ran all cumulative redshift jobs for 20211212, and the cumulative redshift job for tile 7733 on 20211215.
• Created nightqa files.
• Created the tsnr afterburner exposures-*.csv and tiles-*.csv files.

Note that the approach of copying all of the psfs and flats from 12-11 is different than what was done in daily, so it may be worth verifying that this approach is at least as good as daily. My approach here is more "proper" than what was done in daily and we have used this technique elsewhere with success, but If it fails here I can repeat these steps using the approach done in daily with a much shorter turnaround.

Also note I didn't make a "rollup" zcatalog of the redshifts from the 17 tiles. If that is needed, please let me know.

[schlafly]

I can confirm that the redshifts on P9 look much better than those in iron. @ashleyjross , I think this would be over to you to see about how painful incorporating this into the LSS analysis would be.

[schlafly]

Just reviewing this ticket---Anthony demonstrated that getting the right cal files linked in here works. So this ticket now is either about making sure that that also happens for Jura or that this night gets appropriate special treatment in the new per-night cal files.

[akremin]

There is now an override file in place that will automatically do the special calibration handling that I outlined above. I noticed the exposure_table hadn't been updated to mark the bad arcs. I have updated that. This lack of flagging wouldn't have made a difference in this circumstance since the override takes priority over any available data.