Closed mdlpstsci closed 4 years ago
Long overdue status update
First things first, it helps to have a working definition of the commanding overheads. The "theoretical" value for the commanding overheads (δttheo) is contained in the header keyword, DARKTIME, and it is computed using information stored in the unique data log (UDL) for the observation. For a given observation, the commanding overheads may be computed from the header keywords as follows: δttheo = DARKTIME - (FLASHDUR + EXPTIME) where, DARKTIME := total time for dark current to accrue in units of seconds EXPTIME := exposure time in units of seconds FLASHDUR := flash duration in units of seconds
To empirically verify the commanding overheads, we analyze stable, hot pixels in dark frames that are taken as part of the CCD Daily Monitor program. For an arbitray hot pixel in one of these dark frames, we can represent the observed pixel value as a linear combination of the contributing sources (dark current, post-flash, and bias).
p(x,y) = (FLASHDUR+ δtobs + EXPTIME ) dc(x,y) + FLASHDUR fr(x,y) + bias(x,y)
p(x,y) := pixel value at (x, y) in units of electrons dc(x, y) := dark current at (x, y) in units of electrons/second fr(x, y) := flashrate at (x, y) in units of electrons/second bias(x, y) := bias level at (x, y) δtobs := observed commanding overheads EXPTIME := same as above FLASHDUR := same as above
Note that since dark current is an inherent property of the detector, it continually accrues the entire time the detector is operating in ACCUM mode which includes the post-flashing. If the DARKTIME keyword accurately encapsulates all the additional offsets associated with commanding, then δtobs = δttheo.
I've analyzed three types of observations; post-flash full frame, regular full frame, regular subarray. The findings can be summarized as follows: Post-flashed full frame δtpf_ff = δtobs -δttheo = 2.43 seconds
Regular full frame δtff=δtobs -δttheo = 0.21 seconds
Regular subarray δtsub=δtobs -δttheo = 2.45 seconds
Finally, we compute the value for the 4th observation type (post-flashed subarray observations) as, δtpf_sub = (δtpf_ff - δtff) + (δtsub - δtff) = 4.46
As you have stated in your summary, @mdlpstsci, the way we are going to implement this is by adding two additional columns (post-flashed vs unflashed) in the CCDTAB. The post-flashed column will contain the values for δtpf_ff, δtpf_sub, while the unflashed column will contain the values for δtff, and δtsub. The values will be applied as an offset to the DARKTIME keyword already present in the header. So, as an example, suppose we have a post-flashed, full-frame observation, then in the DARKCORR step we will scale the superdark by darktime + δtpf_ff (instead of just darktime) before performing the subtraction.
@nmiles2718 , is this in ISR somewhere?
@nmiles2718 , is this in ISR somewhere?
Yes, but I'm still writing up the ISR and so it has not been published yet. π
@nmiles2718 - a couple of questions.
exposure time
(SBC) or the exposure time + flash duration
which are then scaled by the hard-coded 3 seconds. I have presumed you want me actually to use the DARKTIME keyword from the FITS header, and scale the keyword by the new values?3 second
scaling is only done for Post-SM4, non-BIAS, WFC data.FLASHDUR and FLASHSTA will be the most useful. I am not sure how the commanding overheads change if the post-flashing is ABORTED, or even if they will change at all. I think something along the lines of
if flashdur > 0 and FLASHSTA=="SUCESSFUL":
use post-flash overheads
else:
use appropriate unflashed overheads
Yes, please use the DARKTIME keyword in the FITS header and then add the appropriate offset
The full-frame scaling (post-flashed or not) can be applied to anything post-SM4. The subarray scaling should be applied to subarray frames taken using the new readout configuration. The new configuration was validated in time for use in cycle 24 and the first observation of Cycle 24 was taken on 10/01/2016. I'll talk to Norman about how to proceed for handling subarray cases taken before 10/01/2016, my guess is that we will use the current behavior (without the offset). I'll get back to you with a final answer ASAP
In order to complete this issue, I need to verify or get clarification for the following items:
1) The DARKTIME keyword from the image header is now to be used as the base value for darktime for all cases instead of
(exposure time) for SBC (exposure time + flash_duration) for WFC and HRC
2) Post-flash state is determined by (flashdur > 0 and flash_status=="SUCCESSFUL").
3) The new scaling factors (regardless of unflashed or post-flashed) are to be used for all WFC and HRC full-frame data acquired on or after SM4 (MJD = 54967, 16 May 2009). The scaling is additive.
4) For WFC and HRC full-frame data acquired < SM4, the darktime is the base value of darktime (Item 1) with no scaling.
5) The new scaling factors (regardless of unflashed or post-flashed) are to be used for supported WFC subarray data acquired on or after CYCLE_24 (MJD = 57662, 01 Oct 2016). The scaling is additive.
6) For WFC and HRC subarray data acquired < CYCLE_24, what should be done? There is no discrimination between full-frame and subarray data in the current code.
7) The supported subarrays are (reference Table 7.7 in ACS Instrument Handbook): "WFC1A-2K", "WFC1B-2K", "WFC2C-2K", "WFC2D-2K", "WFC1A-1K", "WFC1B-1K", "WFC2C-1K", "WFC2D-1K", "WFC1A-512", "WFC1B-512", "WFC2C-512", "WFC2D-512", "WFC1-IRAMPQ", "WFC1-MRAMPQ", "WFC2-ORAMPQ", "WFC1-POL0UV", "WFC1-POL0V", "WFC1-SMFL"
I think this is it! Best, Michele
Answers to questions provided below:
DARKTIME
keyword, which is the scaling factor for the DARKCORR
step.
DARKTIME
with the offset added in should be used to scale the superdark in DARKCORR
DARKTIME
value present in the header should be used as the base value of DARKTIME
DARKTIME
keyword and that value should be used to scale the superdark in DARKCORR
.DARKTIME
keyword present in the header without any offset added in.Per the email conversation with resident polarizer expert @tddesjardins, we should also include the 4 other polarizer subarray apertures that are automatically generated by commanding software and are not listed in Table 7.7.
@nmiles2718 1) Can you suggest one dataset per category (six total), perhaps the ones you used for testing for the following: a) FF, flashed and unflashed, pre-SM4 b) FF, flashed and unflashed, post-SM4 c) subarrays, pre-Cycle 24 d) subarrays, post-Cycle 24
@tddesjardins 2) Just to clarify...since the POL and RAMP apertures did not change their names pre- and post-Cycle 24 so they always look as though they are supported subarrays, I should only apply the additive scaling to post-Cycle 24 data?
@mdlpstsci
I have moved a series of test datasets for you to my central store in a directory called forMichele/darktime_test_data
. The only dataset missing is the one containing flashed, subarray data taken before Cycle 24 and this is because there are no data meeting this requirement. I had @tddesjardins double check my search via another route and his yielded the same info.
Full-frame, flashed and unflashed, pre-SM4
flashed:forMichele/darktime_test_data/pre_sm4/postflash
unflashed: forMichele/darktime_test_data/pre_sm4/unflashed
Full-frame, flashed and unflashed, post-SM4
flashed: forMichele/darktime_test_data/post_sm4/postflash
unflashed: forMichele/darktime_test_data/post_sm4/unflashed
Subarrays, pre cycle 24
flashed: forMichele/darktime_test_data/subarray_pre_Cy24/postflash
(DOES NOT EXIST)
unflashed: forMichele/darktime_test_data/subarray_pre_Cy24/unflashed
Subarrays, post cycle 24
flashed: forMichele/darktime_test_data/subarray_post_Cy24/postflash
unflashed: forMichele/darktime_test_data/subarray_post_Cy24/unflashed
@nmiles2718 I have tested the new calacs software using the datasets and preliminary new calibration file you have provided. You can capture the latest calacs changes from GitHub for your examination and testing. I will note that in order to perform my testing, I temporarily modified the calacs code to ignore the gain factor in the CCDTAB for purposes of finding a matching row in the table. Some of the particular test files did not have actual matching rows as the gain factors did not match. I note this for you in case you use the same data for testing and find that calacs does not find a matching row. I assumed the preliminary CCDTAB was just a quickie test file so that I could proceed. If this file is, in fact, very close to the final and new CCDTAB, then some decisions have to be made. If calacs does not find a viable matching row in the CCDTAB, calacs stops processing.
FYI - the factors which must match between the input data and the CCDTAB are: amp, gain, chip, and four ccdoffset values.
Please know I will eventually need the final and real new CCDTAB in order to generate new regression truth files for our test system.
@mdlpstsci Yes, that was my mistake. I only supplied a single CCDTAB for a dataset that uses multiple, distinct CCDTABs generated over the lifetime of the instrument. I'm working through the updates now and testing everything, including creating updated versions of every CCDTAB. After updating the files to use the correct CCDTABs, I was able to process every file in the dataset I generated for you.
For your convenience, my regression analysis may be found here
@mdlpstsci
I have completed my review and everything looks as expected π . Since the updated CCDTABs have two new columns, I have to file a new issue in JIRA for CRDS so that they can make the necessary changes to accept the newly formatted CCDTABs. Once CRDS is ready to accept the new format, I will deliver them to TEST and we can run the new software on the full regression suite.
@nmiles2718 Thanks for the update. I will need to get the new reference files, and hopefully they will have their final names (post-CRDS). I need to update the ACS datasets we use for regression testing and generate new truth files.
On 05 March 2020...
Hi Michele, The new CCDTAB reference files have been delivered to HST TEST. You can find them here: /grp/crds/hst/test/references/hst
The filenames are as follows:
43515536j_ccd.fits 43515537j_ccd.fits 43515539j_ccd.fits 4351553aj_ccd.fits 4351553bj_ccd.fits 4351553dj_ccd.fits 4351553ej_ccd.fits 4351553gj_ccd.fits
Let me know if you need anything else from Nate or myself. As always, keep me updated on how things go. Thanks! Best, Meaghan
@nmiles2718 @mmcdonald123 I have looked at the new reference files, and they are all applicable to the WFC detector only. As this work was proceeding, we never explicitly said the updates do not apply to HRC, and the HRC was included in some discussions on Github Issue 326 (e.g., 07 January). Having said this, it looks like this darktime correction update is really only about WFC. Is this a true statement?
The current public release of CALACS has the CCD darktime correction for HRC as the (exposure_time + flash_duration). Only the WFC got the additional factor of 3.0s for post-SM4 data. If the new darktime correction computation (CCD, pre- and post-SM4, and flashed/unflashed) only applies to WFC, then there need to be some changes to the updated code. The darktime correction is now obtained from the DARKTIME FITS keyword in the science headers. The darktime correction is then modified depending upon unflashed/post-flashed, full_frame/subarray, SM4/Cycle24.
(1) Do you want the darktime correction for HRC to remain as the exposure_time + flash_duration (e.g., the current public release)? If so, I need to make additional small code changes. (2) I have been told that even if you want (1) above, the pipeline is supposed to be data (reference file) driven. As such, there need to be new HRC CCDTAB files with the same new columns as done for WFC. The code which reads the CCDTAB is detector-agnostic, so it needs to be able to read the same reference file formats. These columns could contain zero values. (3) If you do not want (1), you still have to do (2). (4) If you want the HRC to use the DARKTIME FITS keyword from the header, but there is no further correction as is done for WFC by the new code, I have to make a small change in the code.
Fundamentally I need to know the correction which should be applied for the HRC, and new CCDTAB files need to be generated for HRC.
This issue is related to and may well supersede IT #1. In the DARKCORR step of CALACS, the value of the DARKTIME keyword is used to account for additional time that dark current accumulates. It has been determined that depending upon the type of observation (i.e., full frame or subarray, post-flash or no post-flash) there is an additional correction needed (ref. ACS ISR 2018-03). The additional correction values need to be read from the CCDTAB reference file, and the DARKCORR algorithm updated to accommodate new logic for applying the correction.