Reprocess exposures from 202109 that had amps marked as bad due to CTE?

[schlafly]

We marked a lot of data from September 2021 as bad due to CTE on r4A and r1D. These sectors now have software corrections in place. We should turn off marking these amplifiers as bad.

Unfortunately, we already ran iron with these marked as bad, so we don't have these amps in that reduction. We should do a mini-reduction of many of these tiles after unmarking as bad to see if there is any issue.

If we had nightqa of iron we could look at a few sample nights to make sure the correction was working. If it was not in place we should see an offset in the affected regions of the sframe sky residuals.

@julienguy , @akremin , thoughts?

[schlafly]

@sbailey, reraising this issue as one we should think about for the next reduction. I think mechanically this just means turning off whatever bad flagging we did on those amplifiers. I checked that the DESI_SPECTRO_CALIB yaml files have the information needed to turn on the CTE corrections on those amps.

My understanding is that we are masking ~one month of data on r4A and r1D due to this issue, and that we don't have any reason to be doing this anymore---just to say that I think this issue is worth trying to address.

[schlafly]

I checked that r1 and r4 have their OFFCOLS updated in the DESI_SPECTRO_CALIB yaml files such that I expect these petals will now process fine if we stop marking the amps on these petals as bad. I think we only had two bad amps at this time, and that these were those two amps, so I think this is everything that we need to look at now. I also checked the iron reduction from a night in the problematic time period to verify that we were masking these amps as bad at that time in iron, and no others. https://data.desi.lbl.gov/desi/users/raichoor/spectro/iron/nightqa/20211001/tileqa-20211001.pdf

I think the next steps would be:

• update the exposure tables (?) to not mark the amps on r1 and r4 as bad due to CTE.
• reprocess one night (e.g., 20211001) to make sure that it still looks good after removing the bad amp flagging.

I expect this will just work, but the r1 CTE does look pretty bad, but I don't think at the level that we would want to turn on the full z1/z3 CTE code. If we did have to do that we'd probably give up since that requires calibration data (1 s flat field) we weren't taking in 2021.

[akremin]

This should be easy to do. As you say we just remove the BADAMP flags for all the amps in those two cameras.

I am curious, was this data reobserved already? Would this enhance these tiles to higher efftime than typical in the survey?

[schlafly]

These have already been reobserved for the most part. I don't have good enough memory to decide how many MTL updates we did before we realized that we had a problem.

On Tue, Feb 20, 2024, 17:39 akremin @.***> wrote:

This should be easy to do. As you say we just remove the BADAMP flags for all the amps in those two cameras.

I am curious, was this data reobserved already? Would this enhance these tiles to higher efftime than typical in the survey?

— Reply to this email directly, view it on GitHub https://github.com/desihub/desisurveyops/issues/127#issuecomment-1955237940, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK2E4JXQV35ZOPWB6TGZALYUUQZVAVCNFSM6AAAAAA2HZ7ZEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGIZTOOJUGA . You are receiving this because you authored the thread.Message ID: @.***>

[sbailey]

@akremin IIRC you tried to reprocess these and then tripped on desihub/desispec#2161 where DESI_SPECTRO_CALIB had default PSFs with overlapping traces. I just fixed that and updated DESI_SPECTRO_CALIB at NERSC, so this should be ready to try again.

[akremin]

Now that several underlying issues in the pipeline are resolved, it was straight-forward to setup a miniprod and process two example nights 20210926 and 20210927 where I have removed BADAMPS flags and allow all data to be processed using the cte algorithms we now have in place. I selected those two nights because they included both bright and dark tiles.

specprod at NERSC: /global/cfs/cdirs/desi/spectro/redux/cte2021test

QA: https://data.desi.lbl.gov/desi/spectro/redux/cte2021test/nightqa/20210926/nightqa-20210926.html and https://data.desi.lbl.gov/desi/spectro/redux/cte2021test/nightqa/20210927/nightqa-20210927.html

If Eddie or someone else in surveyops wants to sign off on this as looking out, then I will remove the BADAMPS from the rest of the impacted nights and commit them to SVN for future productions.

[schlafly]

@ashleyjross , would you mind glancing at these as well? My sense is that these are good. You can see some CTE-like residuals on r1 at the extreme red edge of the spectra, but they don't seem to be impacting redshifts much. There is a rather small accumulation of sky fibers in both r1 & r4 but it is much less than we see for many more recent CTE issues.

[ashleyjross]

I agree these look fine. It isn't a huge number of tiles and I'm not finding the original QA pages to compare to, though. If there is a night to test with a large number of dark tiles and the original QA page to compare to, that would be great, though it does seem very likely we would sign off.

[schlafly]

I agree these look fine. It isn't a huge number of tiles and I'm not finding the original QA pages to compare to, though. If there is a night to test with a large number of dark tiles and the original QA page to compare to, that would be great, though it does seem very likely we would sign off.

I think this was from before we made nightqa products, and any nightqa products we did have would have the amps masked. Is that what you want to compare against?

Message ID: @.***>

[schlafly]

Anand has the iron nightqa e.g. here: https://data.desi.lbl.gov/desi/users/raichoor/spectro/iron/nightqa/20210927/nightqa-20210927.html

Message ID: @.***>

[ashleyjross]

That comparison helps. I don't see anything obviously getting worse anywhere. It would still be nice to see a night with a large number (>20) of dark tiles.

[schlafly]

@akremin , is it straightforward to also run 20211001? That should have 26 dark tiles.

[akremin]

Yes that is straightforward. I'll submit those today and post again when it's ready to review.

[akremin]

The queue was competitive, but perlmutter was finally able to get through all of the 20211001 jobs. I've run the tsnr and nightqa scripts, so that is now ready for review @schlafly @ashleyjross .

QA: https://data.desi.lbl.gov/desi/spectro/redux/cte2021test/nightqa/20211001/nightqa-20211001.html

[schlafly]

Ugh, no, this was a good test night with a handful of tiles on the bright / dark boundary. My impression is that for r1 we would really want to do the more "advanced" CTE correction but r4 is fine as is. @ashleyjross?

[akremin]

@schlafly 1s flats weren't taken at this time, so my impression is that we can't use your more advanced code.

[schlafly]

Thanks. Yes. So yeah, I think my proposal would be to leave r1 masked but clear the r4 mask, but let's let Ashley take a look too.

[ashleyjross]

I agree it looks like petal 1 still has issues, though small. If this night is the worst of it, we can possibly accept this and mask down-stream if need be. We really need to construct something like a "questionable data" list that flags everything for follow-up...

[schlafly]

Can you confirm that you're happy with r4? I don't like that r1 behavior and would keep it masked, but r4 is completely nominal as far as I'm concerned.

On Thu, Apr 4, 2024, 12:23 ashleyjross @.***> wrote:

I agree it looks like petal 1 still has issues, though small. If this night is the worst of it, we can possibly accept this and mask down-stream if need be. We really need to construct something like a "questionable data" list that flags everything for follow-up...

— Reply to this email directly, view it on GitHub https://github.com/desihub/desisurveyops/issues/127#issuecomment-2037659778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK2E4JLXXTJZJDOIZF7XHLY3V5BPAVCNFSM6AAAAAA2HZ7ZEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZXGY2TSNZXHA . You are receiving this because you were mentioned.Message ID: @.***>

[ashleyjross]

I see a tiny bit of clumpiness in the petal 4 sky z, but nothing at a level that I think we would have flagged at the time and everything else looks good to me. So, yes, I'm happy with petal 4.

[akremin]

If we want more granular coverage, I'm happy to submit all nights in a miniprod and we can assess r1 night-by-night.

Otherwise, I will proceed with unflagging r4 but leaving r1 as flagged.

Let me know which option you'd prefer, Eddie.

[schlafly]

Let's go ahead unflagging r4 but leaving r1 flagged, thanks.

[akremin]

Flags for r4A have been removed, while r1D have been left in the exposure_table's. This was done with SVN revisions 3780 and 3781.

Those changes will automatically be incorporated into future productions. I am assuming we do not want to do anything in daily about this? If so I believe we can close this issue.

[sbailey]

Yes, let's not reprocess daily and focus on future productions. I'm going to close this ticket, but @schlafly feel free to re-open or ping me on Slack if you think we should reprocess daily.

[schlafly]

I agree that we should not reprocess daily.