ACS CTE: Use amp dependent read-noise

dborncamp commented 7 years ago

My understanding of the acscte algorithm is that one of the first steps is to determine if a pixel has enough signal to noise to apply a decent correction. It does this by checking to see if the value of a pixel in the science array is above the readnoise of the detector which is a straight threshold. This is a good check, but it appears that acscte uses a readnoise value that is quoted in the header of the pctetab as the threshold value. However, @nmiles2718 discovered the value in the pctetab is wrong for ACS/WFC and the readnoise changes for each amplifier so a single value is not appropriate for determining this threshold.

@nmiles2718 also discovered that this has a large effect in the number of corrected pixels when CTE correcting low background observations or dark images as there is very little signal. Also, because different amplifiers have different read noise properties it could correct amplifiers differently when it should not. While these pixels may not have much moved charge it does make a difference.

For a solution I think we could do one of three things:

I think that at this point the blv_tmp file error arrays should have the readnoise and bias so we could use that as an indicator of initial signal to noise on a pixel by pixel basis. If the error array has the Poission noise from the science in there we may not want to do this, but we should consult Jay about that.
Use the error array of the bias image appropriately scaled to determine signal to noise also on a pixel by pixel basis.
Use an amplifier dependent readnoise by reading the values in from the ccdtab.

We will probably want to consult with Jay on the best solution.

pllim commented 7 years ago

c/c @JayVB

jamienoss commented 7 years ago

@dborncamp & @nmiles2718 I'm on board and think I know what you are referring to. This is due to the way that it was implemented for WFC3 in that each amp isn't iterated over and corrected individually. For ACS I kept and used all of the amp dependent parameters just as the gen1 version did, however, you are right in that the PCTETAB is the unwanted collective here.

For clarity are we talking about PCTERNOI in the PCTETAB? Are you saying that this value is wrong, full-stop, or that it is only wrong in that it should be amp dependent? My interpretation of your description is that both are true. If the value is wrong what is being done to at least rectify this?

I'm currently working on your last suggestion of using the amp dependent RN from the ccdtab. At least you can play around with it and see how it performs before we make anything concrete.

jamienoss commented 7 years ago

How do I know which row of the ccdtab to use?

I looked at the data handbook but the most info I could find is this and it isn't much if even accurate (doesn't seem to list all columns).

The “CCD Characteristics Table” reference file, recorded in the image header
keyword CCDTAB, is used to determine the bias (for certain subarrays only), gain, and
read noise values for an observation, and for use in calculating error values (σCCD) for
the ERR array. The table columns are
• CCDAMP: every possible configuration of the amplifiers used for readout.
• CCDCHIP: the chip being read out.
• CCDGAIN: the commanded gain.
• CCDBIASA, CCDBIASB, CCDBIASC, CCDBIASD: the commanded bias
level of the amplifiers.
• BINAXIS1, BINAXIS2: the pixel bin sizes.
The CCDTAB reference table uses these commanded values to determine an
observation’s physical readout characteristics that are written to the
flt.fits/flc.fits calibration file. For instance, read noise values for each
amplifier are populated in header keywords READNSEA, READNSEB, READNSEC,
and READNSED; A-to-D gain in ATODGNA, ATODGNB, ATODGNC, and ATODGND;
bias voltage offset values in CCDOFSTA, CCDOFSTB, CCDOFSTC, and CCDOFSTD).

Is there some other reference source I can look at?

I took a look at the actual file and the rows seem to be different amp configurations but it is still not clear to me.

jamienoss commented 7 years ago

Actually, hold on that as I think I see now that the code does this.

jamienoss commented 7 years ago

Yep got it, all sorted.

dborncamp commented 7 years ago

The PCTERNOI keyword in the current pctedab seems to be very wrong, it does not match anything on ACS/WFC. This could be fixed by a delivery of a new file but there is another issue with doing that, the readnoise changes over time for each amplifier where the CTE model does not. The changes are mostly caught by a new CCDTAB so that would be a better place to read in initial readnoise if we are not going to try to read in from the bias or error arrays.

Here is an example of how the ACS readnoise has changed over time for each amplifier: rdnoise_august_label_720

jamienoss commented 7 years ago

@dborncamp @nmiles2718 I haven't been able to test this yet, I currently can't remember where my CTE related data is on my home machine - sorting this out.

It should work though... https://github.com/jamienoss/hstcal/tree/issue213-acs-cte-amp-dep-RN-clipping

Update:

Branch tested, A-OK, c.f. comment

pllim commented 7 years ago

FWIW if a keyword is no longer used in PCTETAB, it should be removed and delivered to HSTDP with the changes here.

nmiles2718 commented 7 years ago

Sorry for the lack of responses, I was out this past Friday. Seems like y'all have everything covered. I would offer to examine that fix, @jamienoss, but I'm still in the process of reprocessing and examining all of the ACS/WFC darks. My brain has been trained on our super darks which will look entirely different than single images that have been CTE corrected because their read noise has dropped by a factor of sqrt(N), where N is the number of images combined to make the super dark. Adding new training data might throw off my learning algorithm as it is not as robust as I'd like...

jamienoss commented 7 years ago

@mdlpstsci this will have to be brought up in the next ACS meeting (I'll tag along for this) and Norman will have to give the green light before we merge. The pipeline team will probably need to be informed if this goes through and a discussion on complete reprocessing should also be conducted.

jamienoss commented 7 years ago

The value of PCTERNOI in the ACS CTE reference file ($jref/16k1747tj_cte.fits) is 3.25. This is the exact same value used in the WFC3 CTE reference file ($iref/zcv2057mi_cte.fits). This value was obviously not checked and updated by the instrument team when it was validated before being handed over to CRDS.

pllim commented 7 years ago

:scream:

jamienoss commented 7 years ago

Note to self:

Look into what needs to be done to make the used values persist beyond the run, i.e. updates to the header for the resultant file(s) etc.

jamienoss commented 7 years ago

@dborncamp @nmiles2718 The branch https://github.com/jamienoss/hstcal/tree/issue213-acs-cte-amp-dep-RN-clipping works, I tested it. You can use this to test out the data diffs etc. This is not a complete code change as I need to see what else may need to be accounted for, e.g. above comment.

dborncamp commented 7 years ago

@jamienoss You are correct in your previous comment, we did not check it. We just assumed that what we got from @JayVB was perfectly correct because we don't fully understand what this correction does. At the very least, we will re-deliver a new PCTETAB with the new darks which will require complete reprocessing anyway.

My expectation is that this will not effect science data much because (as I currently understand it) the read noise is used to see if there is actually signal in the pixel. Almost all science images have a background signal above the readnoise so they should pass this first pass.

Does your new fix read in the readnoise values from the CCDTAB or from the header of the image?

jamienoss commented 7 years ago

Does your new fix read in the readnoise values from the CCDTAB or from the header of the image?

From the CCDTAB.

jamienoss commented 7 years ago

Does your new fix read in the readnoise values from the CCDTAB or from the header of the image?

From the CCDTAB.

Is this data contained within the image header? I took a look and found only this unpopulated section (j6lq01naq_raw.fits):

 / CALIBRATED ENGINEERING PARAMETERS                                                                                                               ATODGNA =             0.000000 / calibrated gain for amplifier A          
ATODGNB =             0.000000 / calibrated gain for amplifier B                
ATODGNC =             0.000000 / calibrated gain for amplifier C                
ATODGND =             0.000000 / calibrated gain for amplifier D               
READNSEA=             0.000000 / calibrated read noise for amplifier A          
READNSEB=             0.000000 / calibrated read noise for amplifier B          
READNSEC=             0.000000 / calibrated read noise for amplifier C          
READNSED=             0.000000 / calibrated read noise for amplifier D          
BIASLEVA=             0.000000 / bias level for amplifier A                     
BIASLEVB=             0.000000 / bias level for amplifier B                     
BIASLEVC=             0.000000 / bias level for amplifier C                     
BIASLEVD=             0.000000 / bias level for amplifier D

This looks to me like a placeholder for these values to updated from the calibration pipeline from the CCDTAB, is this correct?

Please advise - from where should I be obtaining these parameters?

dborncamp commented 7 years ago

Those values should be populated in the blv_tmp.fits files. But I believe that these values are just copied from the CCDTAB.

nmiles2718 commented 7 years ago

I just checked Data Handbook and it looks like these values are populated during the first step of acsccd, doDQI, which initializes the DQ with information from the BPIXTAB. Looking at the output of CALACS, it seems like the information from the CCDTAB is populated just before DQICORR step. So you are correct @jamienoss, the values seem to be pulled from the CCDTAB.

CALACSBEG*** CALACS -- Version 9.2.0 (01-Jun-2017) ***
Begin    18-Sep-2017 12:44:38 EDT

Input    im4caltest_wfc_raw.fits
LoadAsn:  Processing SINGLE exposure
Trying to open im4caltest_wfc_raw.fits...
Read in Primary header from im4caltest_wfc_raw.fits...

CALACSBEG*** ACSCCD -- Version 9.2.0 (01-Jun-2017) ***
Begin    18-Sep-2017 12:44:38 EDT
Input    im4caltest_wfc_raw.fits
Output   im4caltest_wfc_blv_tmp.fits
Trying to open im4caltest_wfc_raw.fits...
Read in Primary header from im4caltest_wfc_raw.fits...
APERTURE WFC
FILTER1 F606W
FILTER2 CLEAR2L
DETECTOR WFC
DESCRIP read in as:

DESCRIP read in as:

DESCRIP read in as:

CCDTAB   jref$uc82140bj_ccd.fits
CCDTAB   PEDIGREE=inflight
CCDTAB   DESCRIP =July 2009

DQICORR  PERFORM
DQITAB   jref$t3n1116nj_bpx.fits
DQICORR  COMPLETE

BIASCORR PERFORM
BIASFILE jref$u3u20559j_bia.fits
BIASFILE PEDIGREE=INFLIGHT 01/09/2009 11/09/2009
BIASFILE DESCRIP =Bias intended for use with full-frame four amp readouts.-----------
BIASCORR COMPLETE

BLEVCORR PERFORM
OSCNTAB  jref$lch1459bj_osc.fits
BLEVCORR PERFORM
Performing stripe removal and bias level subtraction.
     bias level of 4265.22 electrons was subtracted for AMP A.
     bias level of 4076.1 electrons was subtracted for AMP B.
     bias level of 4499.95 electrons was subtracted for AMP C.
     bias level of 4666.73 electrons was subtracted for AMP D.
BLEVCORR COMPLETE

    Uncertainty array initialized,
    readnoise =4.57,3.91,4.25,4.04
    gain =2.02,1.886,2.017,2.011
   default bias levels =4242,4060.6,4481.8,4645.4

spacetelescope / hstcal

ACS CTE: Use amp dependent read-noise #213

Update:

Note to self: