z1 data not trustable for 20231202-12

[araichoor]

z1 was replaced on 2023-11-30, first data since that taken on 20231202. Temperature was changed on 2023-12-12, and timing was changed on 2023-12-13. So, data quality should be good for 20231213 and later nights.

Observations from 20231206-14 have been archived + MTL updated (on Dec. 11 and 15). Those represent 166 dark tiles and 53 bright tiles.

Even though we do not see obvious effects in the redshifts at the night_qa level, we cannot guarantee that those 20231202-12 data are good for LSS analysis.

We decided at today's surveyops telecon to reprocess those data, with marking that petal as bad.

I let @julienguy, @sbailey and others to correct/complete this ticket (apologies for any errors I ve written!).

And I'll provide soon a check on whether those 20231202-12 tiles have been overlapped by observations since 20231215+.

[araichoor]

@sbailey:

Foreword: I provide below:

• the list of 20231202-12 tiles where the petal=1 overlaps any of the 20231213-17 tiles;
• the list of 20231213-17 tiles overlapping the petal=1 of any of the 20231202-12 tiles.

Besides, I am not certain of how that is dealt with the MTL re-processing...

Result: I find the following overlapping tiles between petal=1 of 20231202-12 ("early") and 20231213-20231217 ("late") tiles:

mainbright  found 8/51 early-pet1 tiles overlapping late tiles: [20137, 22819, 25148, 25156, 25235, 25697, 26076, 26077]
mainbright  found 8/47 late tiles overlapping early-pet1 tiles: [20440, 20468, 21002, 21254, 21256, 21381, 21710, 22114]

maindark    found 50/188 early-pet1 tiles overlapping late tiles: [1457, 1492, 1559, 1562, 1803, 1999, 2013, 2032, 2036, 3009, 3049, 3114, 3369, 3518, 3559, 3589, 3961, 3973, 4276, 4617, 4626, 4635, 4829, 4931, 5368, 5483, 5526, 5539, 5543, 5973, 6258, 7097, 7652, 8223, 8247, 8248, 8269, 9306, 9759, 9761, 9777, 10215, 10218, 10238, 10268, 10643, 11403, 11532, 11746, 11785]
maindark    found 45/71 late tiles overlapping early-pet1 tiles: [1264, 1442, 1487, 1497, 1979, 2394, 2408, 2412, 3362, 3801, 3919, 4039, 4530, 4656, 5111, 5118, 5138, 5509, 5537, 5548, 5599, 5839, 6512, 7757, 8057, 8243, 8253, 8265, 8674, 8678, 9077, 9251, 9779, 9837, 10178, 10824, 10854, 10868, 11094, 11341, 11366, 11371, 11372, 11394, 11418]

Method: I start from exposures-daily.csv, and restrict to tiles with exposures with EFFTIME_SPEC>0. Then I consider all the reachable targets:

• by fibers from petal 1 for 20231202-12 tiles;
• by any fiber for 202313+ tiles.

And I flag the targets in common with a boolean column (LATE_OVERLAP or EARLYPET1_OVERLAP).

Here s an example:

One can see for instance that, with this approach,TILEID=8306 (ra, dec = 36.4, 19.5) observed on 20231216 is not overlapped by the petal 1 of TILEID=6258 (ra, dec = 35.5, 18.2) observed on 20231206, but would overlap that tile if we would consider all the ten petals.

As said on slack, that list will likely grow as we continue observing...

Material: Please below the script + files:

raichoor@nid200006:/global/cfs/cdirs/desi/users/raichoor/spectro/daily/z1-20231202-20231212> ll
total 1.8G
-rw-r----- 1 raichoor desi 5.6K Dec 18 21:13 z1-20231202-20231212.py
-rw-rw---- 1 raichoor desi  60M Dec 18 21:13 z1-20231202-20231212-mainbright-early-pet1.fits
-rw-rw---- 1 raichoor desi 576M Dec 18 21:13 z1-20231202-20231212-mainbright-late.fits
-rw-rw---- 1 raichoor desi 228M Dec 18 21:14 z1-20231202-20231212-maindark-early-pet1.fits
-rw-rw---- 1 raichoor desi 905M Dec 18 21:14 z1-20231202-20231212-maindark-late.fits

The fits files are generated with the following (from an interactive node):

python z1-20231202-20231212.py --outroot z1-20231202-20231212-mainbright --faflavor mainbright --numproc 256
python z1-20231202-20231212.py --outroot z1-20231202-20231212-maindark --faflavor maindark --numproc 256

And one can identify e.g. the maindark 20231213+ tiles overlapping the petal=1 of 20231202-12 tiles as follows:

d = Table.read("z1-20231202-20231212-maindark-late.fits")
tileids = np.unique(d["TILEID"][d["EARLYPET1_OVERLAP"]])

ps: my original reported numbers on slack were bugged, sorry...

[araichoor]

for the record, here are the per-petal, per-tracer n(z) for:

• 20231202-12: potentially problematic nights for petal=1;
• 20231213-17: reference set, where petal=1 should be clean.

To gather the data for each tile, I first consider the $DESI_ROOT/spectro/redux/daily/attic/tiles/cumulative/ folder; and, if nothing s there, I look in $DESI_ROOT/spectro/redux/daily/tiles/cumulative/

I restrict to COADD_FIBERSTATUS == 0 fibers, and use the desispec.validate.validredshifts.validate() function to select reliable redshifts. For each petal, I report the fraction of reliable redshifts in the legend, and display the n(z) of those.

For 20231202-12:

For 20231213-17:

I don t see anything standing out in petal=1 for 20231202-12 (though that obviously does not mean that those redshifts are ok for LSS analysis).

That petal=1 kind of tend to have a success rate on the lower-end of the 10 petals distribution for ~all tracers, but that s also the case for 20231213-17. I notice that for the ELG_LOP and QSO, the petal=1 for 20231202-12 has a spike at z=1.33: that s probably the same thing as reported here, where an artefact is interpreted by redrock as the oii doublet at that redshift: https://desisurvey.slack.com/archives/C01HNN87Y7J/p1702597635520259.

[schlafly]

FYI, I think the z=1.33 spike is the only thing that bothers me here. As you say, it's the same story as with the sky redshifts and is the one thing that we had previously expected to be problematic with the z1 redshifts. But it doesn't seem to be leading to QSO misidentification.

i.e., reasonable to post-facto mask these in the MTL. We'll not be observing the highest priority targets that we could but that's not too bad; at least we're not missing LyA or observing lots of garbage LyA.

[schlafly]

I think we've done everything we intend to here, and can close this ticket? Probably with an associated entry in https://github.com/desihub/desisurveyops/blob/main/doc/closed_issues.ecsv ? Do you want to take this, @araichoor , and close the ticket?

[schlafly]

I did this, closing.