Jura bad fibers 551-553 due to z1 CTE edge effects

[sbailey]

Rongpu identified fibers 551,552,553 as having bad ELG N(z) distributions in Jura when they looked fine in Iron. @schlafly this appears to be due to an edge effect in the CTE corrections added for Jura. Doing a deep dive on night=20220102, expid=116442, tileid=1269, camera=z1 comparing the raw extracted flux (jura-iron):

Zooming in around fibers 551-553:

There is a bigger impact on the redshifts for those 3 fibers because smaller ranges of their wavelengths are CTE corrected, leading to steps in the spectra, where the non-CTE-corrected Iron (blue) is more correct than Jura (orange):

It's unclear to me whether the true step is in the original extracted spectra (dominated by sky), or whether there is some additional effect having to do with building a sky model from fibers with and without CTE correction and subtracting a bad sky step.

@schlafly can you dig into the CTE correction for this camera and this edge effect?

[sbailey]

Another note: the difference Jura-Iron also clearly shows the bundle structure of the fibers, but it is odd that on the leading edge it impacts 3 fibers in a "diagonal" way but in all other bundle edges it impacts only the first fiber of each bundle. I think there could be some legitimate bundle structure that appears in the "true" CTE correction, but this smells like there might be a bug/bias/fragility in the first-pass extractions that are used to make the CTE corrections.

[schlafly]

Ugh, I don't think this is going to have a clean fix. FWIW, the first pixel in the correction is fixed in CCD space (i.e., corresponds to a row of CCD pixels on the device). Meanwhile the bundle edges correspond to a fiber, and so aren't fixed in CCD space.

It is true that the first pixel is different from other pixels in that one gets to pick if the trap starts full or empty or what. I guess I have a little flexibility to play around there.

[schlafly]

Looking at this a little, here's a plot of the amplitude and chi^2/dof of the simplified Regnault CTE fits on z1C and z3C in Jura, scraped from spectro/redux/calibnight

Note that before MJD 59800 the z1C correction is particularly bad. Since the correction is also smallish during this time, the easiest option would be to just use the ordinary OFFCOLS CTE treatment then. The plotted night is 20220102, aka MJD 59581, at the end of the first segment of these points.

Other options:

1. Investigate full "Regnault" fitting for this data; maybe that gives better RMS / less bad first pixel offsets. That would "only" take modifying the yaml files and rerunning and inspecting the output.
2. Add an additional parameter to the simplified Regnault treatment that is "how full should I make the first pixel".

My preference is just to revert to OFFCOLS for the early times, but 1. could also turn out to be easy in principle.

[julienguy]

The edge effect in the charge trap correction is also the cause of bad redshift for fiber 3969 (see issue #2283). Fiber 3969 spectra trace is at the beginning of a CTI region caused by a charge trap in z7 amp B. Columns affected: CCD_X=2057:3724 (the fiber is centered at CCD_X=3719).

[akremin]

@schlafly Has time brought any additional clarity on this topic for you? Between using OFFCOLS and the full "Regnault" fitting? Is it worth running a small test with full "Regnault"? Sorry, I know you have a lot of other tasks on your plate at the moment...

[schlafly]

I think we should just mask these, likely only at late times.

[sbailey]

Fibers 551,552,553 have been added as NEARCHARGETRAPFIBERS in spec/sm10/sm10-z.yaml (z1) in desi_spectro_calib. That will propagate as a FIBERSTATUS bit, but they are not otherwise masked/excluded. Closing ticket.