Weird behavior in r1 of TILE 5648 on 20220324; ion pump glow contamination

[rongpu]

EXPID 127342-127345 are affected by this issue, although only 127342 (TILEID 5648) shows noticeable effects on the redshifts.

QA for tile 5648, showing many objects with z=~0.09 in petal 1:

Example of a z=~0.09 spectrum in tile 5648:

[schlafly]

Here's an animation of the preproc files for r1 around 127342. https://data.desi.lbl.gov/desi/users/schlafly/misc/20220324-r1-preproc.gif Clearly r1 goes a bit haywire in 127342 and then works itself out. I don't see anything in the night log. I think @djschlegel might have even been on the call at that point, preparing for the observation of the following LyA tile.

It looks to me like the effect is low-order enough that @julienguy 's inter-bundle fitting code could be used on this one to good effect. That would be another case where we need to turn this code on selectively to a particular set of exposures, like those from 20211226.

Alternatively, we mark this amp as bad and reprocess. This went into the MTLs, so we'd need to go through that process, but that's fine.

[julienguy]

There is indeed an excess of 5 to 10 electron per pixel in r1 expid=127342, which fades away in the next exposures and becomes negligible at expid=127347 and above. There is no excess background in previous exposures. There is no excess in any other camera (including b1,z1) so it's not an external light contamination.

[julienguy]

It might be related to a glitch in the SM10 red cryostat pressure I see in grafana. The spike in pressure is at 03:27 (on 2022-03-25) and the exposure with the highest background (expid=127342) has DATE-OBS=2022-03-25T03:18 and an exposure time of 1012 sec. (see follow-up on slack instops channel if interested)

[julienguy]

(I edited my previous post to correct the value of DATE-OBS and the exposure time, the spike in pressure happened during the exposure)

[schlafly]

Wow, okay. That's pretty definitive.

It's much more interesting to figure out what went wrong, here, but continuing to focus for the moment on how we should treat this particular tile: do you feel like we should try to model it via your preproc code, or mark them as bad (at least the first one)?

[schlafly]

And in case anyone else is interested... the spectrograph pressures are kind of interesting. Here's the last week for a few different cryostats: SM10 (peak is the one under discussion in this issue): Tallest peak in the last week: Different time scales: No idea what anything means, of course!

[julienguy]

I tested running the preprocessing with option --bkgsub-for-science . It does correct most of the effect. There is a residual systematic bias of 1 electron, but that should be acceptable. At least that would save several r1 exposures on the special tile 82636.

So I suggest rerunning r1 for exposures 127342 127343 127344 127345 127346 with the preproc option --bkgsub-for-science.

Example

desi_preproc -i $DESI_SPECTRO_DATA/20220324/00127342/desi-00127342.fits.fz --cam r1 --bias $DESI_SPECTRO_REDUX/daily/calibnight/20220324/biasnight-r1-20220324.fits.gz -o preproc-r1-00127342-bkgsub.fits --bkgsub-for-science

Plot comparing reductions without ('daily' prod) and with this option.

[schlafly]

Great. That means we don't take any actions here on the surveyops side; the issue didn't seem to affect LyA decisions, so we don't need to try to fix the MTL rapidly. But I leave it open with the doesnotblockmtl flag until we devise the scheme to mark this exposure and exposures like it (20211226) as needing --bkgsub-for-science.

[cmvgit]

About the ion pressure spike in SM10RED. 1-/ do not worry about the oscillations of Pion, they are low level and are present on all cryostats (when the ion pressure goes below ~7 to 8e-8 mbar 2-/ There are "electronic" spikes that occur from time to time on the ion pressure, no worry 3-/ The spike reported by Julien in SM10RED 25/3 3h30 UTC is more worrying because it is very high (~3e-5 mbar) and last for a long time (more than 2 hours to recover). Thus it is not related to electronic noise.

• I check all the parameters (LPT power, cryostat temperatures, pressure in the BLUE and NIR SM10 cryostats) -> all is OK. This spike seems to have only affected the pessure, not the thermal nor the LPT. Thus it seems to me unprobable that it could affect the CCD noise level.
• I check the behavior of SM10RED since 3/1/2022 and do not detect another important spike like this one. 4-/ Thus I do not know what has happened. This really looks like some air has entered the cryostat. The only case where we have observed such a behaviour was on the Saclay test bench and it was due to a knock/schok on the cryostat pipe near the isolation valves. Although that seems not likely because it was around 20h30 24/3 Kitt-Peak time. -> let's see if this kind of problem comes back (before changing the cryostat or the ion pump)

[cmvgit]

Sorry I just realize that my github name was not really explicit: cmvgit == Christophe Magneville

[cmvgit]

Same problem on SN2/SM3 NIR:

• 26/3 2h42 UTC == 25/3 19h42 KP and last for ~12h -> may be some dark? Bias? at that time
• Also identify a spike 22/2/2022 11h43 UTC last 2h

[julienguy]

Thanks Christophe. It's actually SM2 NIR which is 'z8'. We have 2 exposures that could have been affected by the pressure spike on 26/3 2h42 UTC : NIGHT=20220325 EXPID=00127448 DATE-OBS=2022-03-26T02:36:49.259056896 (UTC) exptime=720.5 (BRIGHT) NIGHT=20220325 EXPID=00127449 DATE-OBS=2022-03-26T02:51:30.593492992 (UTC) exptime=1342.1 (DARK)

Looking at median over rows in the preprocessed images. I detect a large offset of ~10 electron/pix for the first exposure=00127448 but a small/negligible effect for the second one (expid=00127449).

We should also reprocess 127448 z8 with option --bkgsub-for-science .

[cmvgit]

OK. I was talking about SN2(winlight number)/SM3(cryostats production number Saclay) which is identified SM2 for you. So we are talking of the same cryostat. The first exposure include the transition between quiet/spike behavior. The second one cover the "end of the spike" but the level is not so high (less than 3e-6 mbar). For SM10 same issue: the "continuation of the spike" has a level even lower (less than 1e-6 mbar) less than 720s after the transition Seems that the light is generated just at the transition ?

[schlafly]

These exposures end up in the same class as 20211216, are waiting on a new pipeline feature which lets us specify special processing for a specific range of exposures. @sbailey , is there a desispec issue I should associate these with?

[sbailey]

I just opened desihub/desispec#1740 to track the need for automated hooks for custom processing of some cameras/exposures. We don't have a good solution even designed yet, much less implemented.

[schlafly]

I don't expect us to turn on special processing here any longer, so we should remove this from the MTL?

[geordie666]

Note there are two options:

(1) Remove this tile entirely from the MTL (the option we use if no overlapping tiles have been subsequently observed).

(2) Change the redshift/quality/classification information for the tile and reprocess the MTL as if we'd observed the tile with the different information.

The approach for option (2) is:

• Stephen (or someone similar) re-archives the tile with the new information from the spectroscopic pipeline. This produces a second (or third, etc.) "archived" directory with a new ARCHIVEDATE for the tile and adds the new ARCHIVEDATE in the MTL tiles file.
• I execute run_mtl_loop with the --reprocess option. This magically reverts all of our previous choices and replaces them with the updated choices.

[schlafly]

If we do (2) but no tiles have overlapped, is that equivalent to (1)?

[geordie666]

It's not really equivalent. In (1) all record of the tile in the MTL ledgers will completely vanish. In (2) we'll keep a careful record of the tile but we'll change the results arising from that tile.

(1) is easier (it's as if we just never reduced the tile, from an MTL perspective). (2) is more honest (it notes we reduced the tile and updated our choices). But, it's more work because the spectroscopic pipeline has to produce new outputs for the tile (and re-archive the tile with those new outputs).

In the past, we've used (1) when we didn't overlap the tile and (2) --- because we really had to --- when we did overlap the tile.

[schlafly]

Sorry, right, the MTLs will not have identical rows. But just making sure I'm understanding, they're equivalent in the sense that what we actually target for DESI will be the same in both cases?

In this case where we have already archived the tile and the tile is not completely bad, we'd need to reprocess & rearchive either way?

[geordie666]

Right, I believe both those statements are true for tiles that haven't already been overlapped by a subsequent tile.

Once we have observed a subsequent overlapping tile, we also have to reprocess and rearchive, because simply removing the tile (1) does not self-consistently alter subsequent decisions in overlapping tiles like (2) does.

Note that, if you aren't sure whether the tile has been overlapped, we also wrote desitarget.mtl() code to check that.

[schlafly]

Okay, let's mark r1 as bad on 20220324, expid 127342-127345, and reprocess. This is tileid (5648, 82636). tileid 5648 needs to be reprocessed potentially in some kind of special mode so that zmtl's get created for P1 and can be used for the MTL reprocessing; I think Stephen is reviewing what we did there? Some mixture of @sbailey , @abhi0395 , @marcelo-alvarez ?

[schlafly]

Pinging this ticket.

[schlafly]

Pinging this ticket. We need to mark r1 as bad on 20220324, expid 127342-127345, and reprocess. Then it will need to go to Stephen / Adam for rearchiving and including in the MTL.

[abhi0395]

@schlafly Re-processed except tile 82636. tsnr+nightqa updated. Some issue with 82636 (ticket (#134) is mentioned describing the problem)

[schlafly]

Just noting for myself, 127342 was on a main survey tile, while 127343-127350 were on 82636, some special program tile. 82636 is the problematic one, so we're missing a chunk of the night in the nightqa, but that's fine.

This is a a redo-MTL case, where we should rearchive 5648 with the new processing. I don't think I need to do anything with the QA here, which is marked as good. @sbailey , to you?

[sbailey]

@schlafly @geordie666 I have rearchived tile 5648 with

desi_archive_tilenight -n 20220324 -t 5648 --badpetals 1

Passing off to @geordie666 for MTL update.

Tile 82636 remains unprocessed due to issue #134 .

[geordie666]

Tile 5648 has been re-updated in the MTL ledgers.