Check N(z) for all fibers overlapping the edge of a CTE correction

[sbailey]

Issue #2286 identified fibers 551,552,553,3969 as having N(z) problems due to imperfect CTE corrections at the start of a charge trap.

Identify and investigate all other cases of fibers overlapping the start of a CTE correction to see if they should also be flagged (see issue #2304 about new FIBERSTATUS bits), and/or add them to #2286 as impacted fibers in case there is an algorithmic fix to the CTE correction in the future.

@rongpu @julienguy @schlafly

[rongpu]

I have identified the fibers that have CTECOLS correction. The CTE columns from grepping CTECOLS in $DESI_SPECTRO_CALIB/spec/*/*.yaml, and their corresponding fibers are listed below. There are 491 fibers in total.

z1 sm10/sm10-z.yaml: CTECOLSC: '543:2057' -> fibers 551-749 z3 sm6/sm6-z.yaml: CTECOLSC: '1668:2057' -> fibers 1705-1755 r6 sm7/sm7-r.yaml: CTECOLSA: '1905:2057' -> fibers 3234-3253 r7 sm8/sm8-r.yaml: CTECOLSD: '2056:3406' -> fibers 3750-3927 z7 sm8/sm8-z.yaml: CTECOLSB: '2057:3724' -> fibers 3750-3970

FWIW, I also created CCD-sized images that map the pixel indices to fibers (I set each fiber to be 5 pixels wide) that may be useful for things like this: /global/cfs/cdirs/desicollab/users/rongpu/useful/desi_ccd_fiber_map_20230423/ccd_fibermap_*.fits.gz For example, to identify the fibers affected by the z7 CTE:

camera = 'z7'
fm = fitsio.read('/global/cfs/cdirs/desicollab/users/rongpu/useful/desi_ccd_fiber_map_20230423/ccd_fibermap_{}.fits.gz'.format(camera))
print("Fibers:", np.unique(fm[:, 2057:3724]))

[rongpu]

I went through the LRG N(z) and z vs time plots for the above fibers, and I only found two fibers that look problematic and not already identified.

The two fibers, 3892 and 3893, have a handful of z>4 redshifts since Jan 2024:

The inspection was done not super carefully, and if anyone wants to double check for bad fibers: https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/per_fiber_qa/jura/per_fiber_redshifts_bgs_any_zcatalog.pdf https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/per_fiber_qa/jura/per_fiber_redshifts_lrg_zcatalog.pdf

[sbailey]

Thanks @rongpu . The ones that we want to check extra carefully are the first and last fiber in each group, and those for which fiber%25 == 0.

Those two new ones (3892 and 3893) are nice examples of a big effect for a short (recent) time period.

[sbailey]

Fibers 551,552,553, and 3969 have been flagged as NEARCHARGETRAP. Others are tracked in new spreadsheet for analysis flags.