Comparing Y1 (iron) and Y3 (jura) redshifts

[rongpu]

In order to separate the purely software issues from the new hardware issues in the jura data quality checks/investigations, here I have done a direct redshift comparison between Y1 (iron) and Y3 (jura). I am the cumulative zcatalogs, and I match both FIBER and TILEID so that the redshifts have the exact same input spectra. I require FIBERSTATUS==0 and EFFTIME_LRG>800 (or EFFTIME_BGS>160 for BGS), but otherwise no redshift quality cuts are applied.

I define a redshift outlier ("n_fail" and "frac_fail" in the tables below) as any object whose redshifts in the two catalogs differ by more than 0.003333*(1+Z_y1), or 1000 km/s.

BGS_ANY: Average outlier rate: 0.18% 17 fibers with >1.5% outlier rate:

FIBER n_tot n_fail frac_fail (%)
  466  1009     22           2.2
  552  1457     53           3.6
  553  1507     50           3.3
  650  1526     27           1.8
  651   798     16           2.0
  675  1487     34           2.3
  700  1543     29           1.9
  725  1326     21           1.6
 1008  1522    388          25.5
 1474  1571     50           3.2
 3038  1542     67           4.3
 3994  1638     61           3.7
 4507   739     13           1.8
 4733   894     19           2.1
 4833  1475     26           1.8
 4891  1074     36           3.4
 4916  1547     44           2.8

LRG: Average outlier rate: 0.20% 19 fibers with >2% outlier rate:

FIBER n_tot n_fail frac_fail (%)
  466   229     10           4.4
  551   603     23           3.8
  552   580     80          13.8
  553   583     43           7.4
  566   538     13           2.4
  625   623     19           3.0
  650   613     38           6.2
  675   601     36           6.0
  700   623     26           4.2
  718   566     12           2.1
  725   370     15           4.1
  961   530     20           3.8
 1008   572    193          33.7
 3038   596     23           3.9
 3994   634     27           4.3
 4321   685     84          12.3
 4720   461     10           2.2
 4833   586     14           2.4
 4891   310     12           3.9

ELG: Average outlier rate: 4.20% 17 fibers with >10% outlier rate:

FIBER n_tot n_fail frac_fail (%)
    0  1000    108          10.8
  551  1271    197          15.5
  552  1226    304          24.8
  553  1326    240          18.1
  566  1022    107          10.5
  625  1261    225          17.8
  650  1269    212          16.7
  675  1299    246          18.9
  700  1275    218          17.1
  725   799    122          15.3
 1008  1206    688          57.0
 2183  1177    152          12.9
 2761  1256    131          10.4
 3038  1282    168          13.1
 4321  1259    173          13.7
 4451   500     56          11.2
 4833  1252    132          10.5

The histograms of the per-fiber outlier fractions with the threshold marked by the dashed line:

[ashleyjross]

The worst one, 1008, was masked from Y1 analysis (list here https://data.desi.lbl.gov/desi/survey/catalogs/Y1/LSS/iron/unique_badfibers.txt). (Does anyone remember if we diagnosed its issue?) Others showing up here that were masked from Y1 include: 466, 651, 3038, 3994, 4321, 4507, 4733, 4891 Is it possible that these have actually been made better in Jura?

[rongpu]

There is certainly a significant overlap with the issues that we found in iron. We did attempt to fix 1008 and 3994 in iron, but apparently it did not work (see https://github.com/desihub/desispec/issues/1946#issuecomment-1418297686) and I don't think we ever looked into them in iron.

[ashleyjross]

Some complementary plots are here (where what was masked in Y1 has Xs on the points): https://data.desi.lbl.gov/desi/survey/catalogs/DA2/LSS/jura-v1/plots/

[sbailey]

Individual fiber issues have been handled in other tickets. Closing this one for Kibo.