Study overscan subtraction problems

[sbailey]

Study the overscan subtraction of CCD raw data as part of the preprocessing. e.g. 20190621/00016474 z0 has a clear step between amplifiers that might be an algorithmic robustness problem: and amps C and D of 20190306/00001076 z2 have row-to-row variations that look like we need row-by-row subtraction (or maybe better bias image):

Track these down and see if any algorithmic updates are needed, or otherwise feedback to the hardware team if it appears to be a raw data problem.

[profxj]

I am digging in on this.

[profxj]

I believe the first issue as regards overscan subtraction has been mititaged by PR #793. But, we need to re-characterize the gain of z0 to confirm (I believe the values will change). Camera z2 is looking good.

See discussion in PR #793 on the second issue raised here.

[sbailey]

What's the status of the z0 gain updates? desispec was updated at KPNO and monitoring restarted after PR #793 was merged, but we're still getting amplifier jump problems on z0, e.g. 20190625/00016618:

[sbailey]

Regarding this comment from PR #793:

The second issue raised in #792 is partly mitigated by doing a by-row removal of the overscan_col region, but I agree with @julienguy that the extra noise incurred is not worth the slight improvement in appearance.

Could you quantify the size of the residual wiggles vs. the amount of noise introduced by doing row-by-row overscan subtraction instead of 2D overscan median scalar subtraction?

[profxj]

Have just finished analyzing new gains for SP0.
This has fixed the Issue with 20190621/00016474 (see image) and presumably any other SP0 images.

[profxj]

Indeed, 20190625/00016618 now looks 'perfect'

[sbailey]

I'm not sure the changes in PR #793 to add column overscan subtraction were beneficial, vs. trading one problem for another. I'm now seeing lots of vertical stripes. e.g. FLAVOR='zero' exposure 20190627/00016727 when processed with desispec master: vs. desispec tag 0.29.0:

The vertical stripes have RMS ~0.6 counts from one column to the next, which is much larger than the readnoise/sqrt(2000), but is consistent with the systematic noise that would be introduced by doing a column-by-column overscan subtraction, given that those column overscan regions are themselves noisy and fairly small.

[profxj]

I have previously used savgol filtering or a bspline fit to beat down the noise. I suggest the same here. And probably the same for overscan_col.

[julienguy]

oh, I thought we were subtracting the average of the overscan rows band, like we do for the overscan columns band. We have to fix this.

[profxj]

You should examine how much overscan_rows varies across the detector before describing using the average as a "fix".

[profxj]

I've explored a bit further and my "expert" opinion is to:

1. Use 1 value for the overscan_columns
2. Collapse the overscan_row using the _overscan() method in preproc
3. Filter that with a savgol filter and go forth

This Notebook shows my recent exploration:

https://github.com/desihub/desispec/blob/overscan/doc/nb/Overscan.ipynb

If we are good with this approach I can proceed to implement it.

[sbailey]

This looks better than either of our previous options, but Julien and Armin are at KPNO this week tuning CCD readout parameters. Let's wait until they are done and then re-evaluate on their final datasets to make sure we are solving a problem that we still have.

[profxj]

Roger

[sbailey]

Please study this on night 20190718 exposures 17335 - 17374 which are flavor=ZERO exposures with spectrographs 0-5 with the updated CCD timing parameters.