Check negative dark image values

[jburkepile]

New camera correction darks are negative

[mgalloy]

Waiting for Alfred.

[jburkepile]

This began with an email from Giuliana about the negative values of the camera-corrected darks in KCor: On Dec 12, 2017, at 11:49 AM, Giuliana de Toma detoma@ucar.edu wrote:

Hi Alfred, thank you very much for taking a look at the KCor calibration data. You can look for example at the day before the eclipse. The calibration file is: /hao/mlsodata1/Data/KCor/calib_files/20170820_190204_kcor_cal_v2_1.4.112.5ms.ncdf the raw images are in /hao/mlsodata2/Data/KCor/raw/2017/20170820/level0 The calibrated data look okay and very possibly the dark and flat are also okay. I was just worried when I saw that almost all pixels were negative in the dark. The median value is about -1500 for camera0 and -1700 for camera1. They used to be around +2000. Only a few pixels, 50 in canera0 and 20 in camera1, are positive. Are they hot pixels? Also, the gain after is dark corrected (see attached image) has a band and an asymmetry left and right. I do not remember seen anything like that in the other cameras. Are they known defects in the cameras? We just wanted to make sure this is okay. This pattern is common to all calibration files, not just to this particular day. Thanks again for taking a look at it. Giuliana -------------------------------- Alfred's response in an email 12/28/17 to Giuliana, Mike and me: Sorry it took me a while to get to this.

This all looks fine to me.

After applying the camera correction the darks are mostly negative. My best guess is that this is because the cameras are cooler at MLSO than they were in the lab and dark current is lower. The spread in the original dark frames is much wider than the corrected dark, see the attached figure, where white is the corrected dark, and red is the original dark. This is a good indication that the camera correction is working as designed, since it is supposed to correct for sensor pixel-to-pixel variation that will spread the distribution of values out. You can also see that the grainy pattern in the image disappears after applying the correction, which is also a good indication that things are working properly

Another good sanity check is to look at a single data file and see what happens. I took 20170820_180008_kcor.fts.gz camera 0 frame 0, and applied the camera correction. Histograms of pixel values basically show the same behavior. After correction (white) there are a lot of pixels with negative values. After dark correction this situation unsurprisingly improves dramatically (green)

There are still some pixels with negative values. Where are they? This is a mask

of those pixels. So it’s some occulter edge effects and pixels outside the FOV. The pixels out of the FOV are probably an indication that the camera warmed up between 18UT and 19UT and the dark current in the dark calibration frames was slightly higher, and thus there is an overcorrection to the data at 18UT. In this case I’m seeing an average of about -90 DN in the regions outside the FOV, compared to data values of ~5000, so roughly 2%. This kind of stuff is why absolute photometry is difficult. I suppose we could consider averaging the pixels outside the FOV and demanding that that average is zero by applying an offset, but I think this is risky because there’s also stray light out there that may not be uniform over the FOV. A better option, if we want to correct this, would be to take darks more frequently and then interpolate the offset. But realistically this is a lot of work and not really worth the effort, since the residual is effectively subtracted out in the polarimetric demodulation on 15s timescales anyway. About the band in the data. This exists in both cameras at the same location. Nowhere in the data reduction do I do anything preferential with these pixels, so the fact that it is there means that it’s something inherent to the sensor/camera. I can see it in the raw dark frames as well. I can’t convince myself I can see it in raw science data, or in raw flats. But looking at the science data example after applying the camera correction, subtracting the dark, and dividing by the gain, I see no trace of it, so that makes me believe that the calibration is right.

I think the issue with the gain as Giuliana shows it is that the gain is already dark corrected, so you’re applying the dark correction twice. The gain is not just an average of flat calibration frames. The flat calibration frames have the camera correction applied, have the dark subtracted, and then are scaled so that (data - dark) / gain gives numbers calibrated against the diffuser intensity. So “gain” is exactly that: the gain of the system from some physical unit to DN.

Lastly, about bad pixels. The white specs are indeed bad pixels - either hot pixels, or maybe pixels that had a blemish on them during the calibration of the camera. There are also unusable bands at the two edges of the image. We did our best to remove as much dust on the sensor as we could during calibration. Looks like we were more successful with the R camera (2 pixels) than with the T camera (49 pixels). While they look bad, 49 pixels is still a minute fraction of the total number of pixels, or even the number of pixels in the FOV. Indeed these pixels need to be handled separately, probably by interpolation of the surrounding area. Most are single-pixel defects, but there are also a few that are larger areas. The camera correction NetCDF file includes a “Bad Pixel Mask” that shows which pixels are not usable.

Cheers, Alfred