CALACS to use the value of the DARKTIME keyword, moderated by the type of observation

[mdlpstsci]

This issue is related to and may well supersede IT #1. In the DARKCORR step of CALACS, the value of the DARKTIME keyword is used to account for additional time that dark current accumulates. It has been determined that depending upon the type of observation (i.e., full frame or subarray, post-flash or no post-flash) there is an additional correction needed (ref. ACS ISR 2018-03). The additional correction values need to be read from the CCDTAB reference file, and the DARKCORR algorithm updated to accommodate new logic for applying the correction.

[nmiles2718]

Long overdue status update

First things first, it helps to have a working definition of the commanding overheads. The "theoretical" value for the commanding overheads (δt_theo) is contained in the header keyword, DARKTIME, and it is computed using information stored in the unique data log (UDL) for the observation. For a given observation, the commanding overheads may be computed from the header keywords as follows: δt_theo = DARKTIME - (FLASHDUR + EXPTIME) where, DARKTIME := total time for dark current to accrue in units of seconds EXPTIME := exposure time in units of seconds FLASHDUR := flash duration in units of seconds

To empirically verify the commanding overheads, we analyze stable, hot pixels in dark frames that are taken as part of the CCD Daily Monitor program. For an arbitray hot pixel in one of these dark frames, we can represent the observed pixel value as a linear combination of the contributing sources (dark current, post-flash, and bias).

p(x,y) = (FLASHDUR+ δt_obs + EXPTIME ) dc(x,y) + FLASHDUR fr(x,y) + bias(x,y)

p(x,y) := pixel value at (x, y) in units of electrons dc(x, y) := dark current at (x, y) in units of electrons/second fr(x, y) := flashrate at (x, y) in units of electrons/second bias(x, y) := bias level at (x, y) δt_obs := observed commanding overheads EXPTIME := same as above FLASHDUR := same as above

Note that since dark current is an inherent property of the detector, it continually accrues the entire time the detector is operating in ACCUM mode which includes the post-flashing. If the DARKTIME keyword accurately encapsulates all the additional offsets associated with commanding, then δt_obs = δt_theo.

I've analyzed three types of observations; post-flash full frame, regular full frame, regular subarray. The findings can be summarized as follows: Post-flashed full frame δt_{pf_ff} = δt_obs -δt_theo = 2.43 seconds

Regular full frame δt_ff=δt_obs -δt_theo = 0.21 seconds

Regular subarray δt_sub=δt_obs -δt_theo = 2.45 seconds

Finally, we compute the value for the 4th observation type (post-flashed subarray observations) as, δt_{pf_sub} = (δt_{pf_ff} - δt_ff) + (δt_sub - δt_ff) = 4.46

As you have stated in your summary, @mdlpstsci, the way we are going to implement this is by adding two additional columns (post-flashed vs unflashed) in the CCDTAB. The post-flashed column will contain the values for δt_{pf_ff}, δt_{pf_sub}, while the unflashed column will contain the values for δt_ff, and δt_sub. The values will be applied as an offset to the DARKTIME keyword already present in the header. So, as an example, suppose we have a post-flashed, full-frame observation, then in the DARKCORR step we will scale the superdark by darktime + δt_{pf_ff} (instead of just darktime) before performing the subtraction.

[pllim]

@nmiles2718 , is this in ISR somewhere?

[nmiles2718]

@nmiles2718 , is this in ISR somewhere?

Yes, but I'm still writing up the ISR and so it has not been published yet. 😅

[mdlpstsci]

@nmiles2718 - a couple of questions.

1. In the ACS/WFC image I have, there are three keywords associated with "flash", FLASHDUR (expo time), FLASHCUR (post-flash current, string=OFF|LOW|MED|HIGH), and FLASHSTA (status, string=SUCCESSFUL, ABORTED, NOT PERFORMED). I need to determine whether or not the image is post-flashed or unflashed, and I have been debating with myself which keyword (or keywords) is best used for this. Suggestions?
2. Despite what has been stated, the DARKTIME keyword is, in fact, not currently used for the dark time correction. The darktime variable is either the exposure time (SBC) or the exposure time + flash duration which are then scaled by the hard-coded 3 seconds. I have presumed you want me actually to use the DARKTIME keyword from the FITS header, and scale the keyword by the new values?
3. Are there limitations on when the scaling should be applied? Currently, the 3 second scaling is only done for Post-SM4, non-BIAS, WFC data.

[nmiles2718]

1. FLASHDUR and FLASHSTA will be the most useful. I am not sure how the commanding overheads change if the post-flashing is ABORTED, or even if they will change at all. I think something along the lines of

   if flashdur > 0 and FLASHSTA=="SUCESSFUL": 
    use post-flash overheads 
   else: 
    use appropriate unflashed overheads

2. Yes, please use the DARKTIME keyword in the FITS header and then add the appropriate offset

3. The full-frame scaling (post-flashed or not) can be applied to anything post-SM4. The subarray scaling should be applied to subarray frames taken using the new readout configuration. The new configuration was validated in time for use in cycle 24 and the first observation of Cycle 24 was taken on 10/01/2016. I'll talk to Norman about how to proceed for handling subarray cases taken before 10/01/2016, my guess is that we will use the current behavior (without the offset). I'll get back to you with a final answer ASAP

[mdlpstsci]

In order to complete this issue, I need to verify or get clarification for the following items:

1) The DARKTIME keyword from the image header is now to be used as the base value for darktime for all cases instead of

(exposure time) for SBC (exposure time + flash_duration) for WFC and HRC

2) Post-flash state is determined by (flashdur > 0 and flash_status=="SUCCESSFUL").

3) The new scaling factors (regardless of unflashed or post-flashed) are to be used for all WFC and HRC full-frame data acquired on or after SM4 (MJD = 54967, 16 May 2009). The scaling is additive.

4) For WFC and HRC full-frame data acquired < SM4, the darktime is the base value of darktime (Item 1) with no scaling.

5) The new scaling factors (regardless of unflashed or post-flashed) are to be used for supported WFC subarray data acquired on or after CYCLE_24 (MJD = 57662, 01 Oct 2016). The scaling is additive.

6) For WFC and HRC subarray data acquired < CYCLE_24, what should be done? There is no discrimination between full-frame and subarray data in the current code.

7) The supported subarrays are (reference Table 7.7 in ACS Instrument Handbook): "WFC1A-2K", "WFC1B-2K", "WFC2C-2K", "WFC2D-2K", "WFC1A-1K", "WFC1B-1K", "WFC2C-1K", "WFC2D-1K", "WFC1A-512", "WFC1B-512", "WFC2C-512", "WFC2D-512", "WFC1-IRAMPQ", "WFC1-MRAMPQ", "WFC2-ORAMPQ", "WFC1-POL0UV", "WFC1-POL0V", "WFC1-SMFL"

I think this is it! Best, Michele

[nmiles2718]

Answers to questions provided below:

1. 👍
2. 👍
3. Scaling typically implies a multiplicative factor, so let’s call it an offset or an adjustment. This is a minor thing but could lead to a lot of confusion because we are adding an offset to the DARKTIME keyword, which is the scaling factor for the DARKCORR step.
   • But yes, the DARKTIME with the offset added in should be used to scale the superdark in DARKCORR
4. Yes, the DARKTIME value present in the header should be used as the base value of DARKTIME
5. Yes, the offsets only apply to the supported subarray modes as of Oct 1. 2016. The offsets should be added to the DARKTIME keyword and that value should be used to scale the superdark in DARKCORR.
6. Subarrays taken prior to that date (flashed and unflashed) should just use the value of the DARKTIME keyword present in the header without any offset added in.
7. 👍

[nmiles2718]

Per the email conversation with resident polarizer expert @tddesjardins, we should also include the 4 other polarizer subarray apertures that are automatically generated by commanding software and are not listed in Table 7.7.

• WFC1-POL60V, WFC1-POL60UV, WFC1-POL120V, WFC1-POL120UV

[mdlpstsci]

@nmiles2718 1) Can you suggest one dataset per category (six total), perhaps the ones you used for testing for the following: a) FF, flashed and unflashed, pre-SM4 b) FF, flashed and unflashed, post-SM4 c) subarrays, pre-Cycle 24 d) subarrays, post-Cycle 24

@tddesjardins 2) Just to clarify...since the POL and RAMP apertures did not change their names pre- and post-Cycle 24 so they always look as though they are supported subarrays, I should only apply the additive scaling to post-Cycle 24 data?

[nmiles2718]

@mdlpstsci

I have moved a series of test datasets for you to my central store in a directory called forMichele/darktime_test_data. The only dataset missing is the one containing flashed, subarray data taken before Cycle 24 and this is because there are no data meeting this requirement. I had @tddesjardins double check my search via another route and his yielded the same info.

Full-frame, flashed and unflashed, pre-SM4 flashed:forMichele/darktime_test_data/pre_sm4/postflash
unflashed: forMichele/darktime_test_data/pre_sm4/unflashed

Full-frame, flashed and unflashed, post-SM4 flashed: forMichele/darktime_test_data/post_sm4/postflash unflashed: forMichele/darktime_test_data/post_sm4/unflashed

Subarrays, pre cycle 24 flashed: forMichele/darktime_test_data/subarray_pre_Cy24/postflash (DOES NOT EXIST) unflashed: forMichele/darktime_test_data/subarray_pre_Cy24/unflashed

Subarrays, post cycle 24 flashed: forMichele/darktime_test_data/subarray_post_Cy24/postflash unflashed: forMichele/darktime_test_data/subarray_post_Cy24/unflashed

[mdlpstsci]

@nmiles2718 I have tested the new calacs software using the datasets and preliminary new calibration file you have provided. You can capture the latest calacs changes from GitHub for your examination and testing. I will note that in order to perform my testing, I temporarily modified the calacs code to ignore the gain factor in the CCDTAB for purposes of finding a matching row in the table. Some of the particular test files did not have actual matching rows as the gain factors did not match. I note this for you in case you use the same data for testing and find that calacs does not find a matching row. I assumed the preliminary CCDTAB was just a quickie test file so that I could proceed. If this file is, in fact, very close to the final and new CCDTAB, then some decisions have to be made. If calacs does not find a viable matching row in the CCDTAB, calacs stops processing.

FYI - the factors which must match between the input data and the CCDTAB are: amp, gain, chip, and four ccdoffset values.

Please know I will eventually need the final and real new CCDTAB in order to generate new regression truth files for our test system.

[nmiles2718]

@mdlpstsci Yes, that was my mistake. I only supplied a single CCDTAB for a dataset that uses multiple, distinct CCDTABs generated over the lifetime of the instrument. I'm working through the updates now and testing everything, including creating updated versions of every CCDTAB. After updating the files to use the correct CCDTABs, I was able to process every file in the dataset I generated for you.

For your convenience, my regression analysis may be found here

[nmiles2718]

@mdlpstsci

DARKTIME review

I have completed my review and everything looks as expected 👍 . Since the updated CCDTABs have two new columns, I have to file a new issue in JIRA for CRDS so that they can make the necessary changes to accept the newly formatted CCDTABs. Once CRDS is ready to accept the new format, I will deliver them to TEST and we can run the new software on the full regression suite.

[mdlpstsci]

@nmiles2718 Thanks for the update. I will need to get the new reference files, and hopefully they will have their final names (post-CRDS). I need to update the ACS datasets we use for regression testing and generate new truth files.

[mdlpstsci]

On 05 March 2020...

Hi Michele, The new CCDTAB reference files have been delivered to HST TEST. You can find them here: /grp/crds/hst/test/references/hst

The filenames are as follows:

43515536j_ccd.fits 43515537j_ccd.fits 43515539j_ccd.fits 4351553aj_ccd.fits 4351553bj_ccd.fits 4351553dj_ccd.fits 4351553ej_ccd.fits 4351553gj_ccd.fits

Let me know if you need anything else from Nate or myself. As always, keep me updated on how things go. Thanks! Best, Meaghan

[mdlpstsci]

@nmiles2718 @mmcdonald123 I have looked at the new reference files, and they are all applicable to the WFC detector only. As this work was proceeding, we never explicitly said the updates do not apply to HRC, and the HRC was included in some discussions on Github Issue 326 (e.g., 07 January). Having said this, it looks like this darktime correction update is really only about WFC. Is this a true statement?

The current public release of CALACS has the CCD darktime correction for HRC as the (exposure_time + flash_duration). Only the WFC got the additional factor of 3.0s for post-SM4 data. If the new darktime correction computation (CCD, pre- and post-SM4, and flashed/unflashed) only applies to WFC, then there need to be some changes to the updated code. The darktime correction is now obtained from the DARKTIME FITS keyword in the science headers. The darktime correction is then modified depending upon unflashed/post-flashed, full_frame/subarray, SM4/Cycle24.

(1) Do you want the darktime correction for HRC to remain as the exposure_time + flash_duration (e.g., the current public release)? If so, I need to make additional small code changes. (2) I have been told that even if you want (1) above, the pipeline is supposed to be data (reference file) driven. As such, there need to be new HRC CCDTAB files with the same new columns as done for WFC. The code which reads the CCDTAB is detector-agnostic, so it needs to be able to read the same reference file formats. These columns could contain zero values. (3) If you do not want (1), you still have to do (2). (4) If you want the HRC to use the DARKTIME FITS keyword from the header, but there is no further correction as is done for WFC by the new code, I have to make a small change in the code.

Fundamentally I need to know the correction which should be applied for the HRC, and new CCDTAB files need to be generated for HRC.