search

spacetelescope / drizzlepac

AstroDrizzle for HST images.
https://drizzlepac.readthedocs.io
BSD 3-Clause "New" or "Revised" License
52 stars 38 forks source link

AstroDrizzle - Investigate new option for final_fillval when WHT=0 #1613

Closed stscijgbot-hstdp closed 7 months ago

stscijgbot-hstdp commented 1 year ago

Issue HLA-1061 was created on JIRA by Jennifer Mack:

[Readthedocs|[https://drizzlepac.readthedocs.io/en/latest/astrodrizzle.html]] states two options for the parameter : final_fillval   float (Default = None)

The value for this parameter is to be assigned to the output pixels that have zero weight or which do not receive flux from any input pixels during drizzling.

If the default of None is used, and if the weight in both the input and output images for a given pixel are zero, then the output pixel will be set to the value it would have had if the input had a non-zero weight.

Otherwise, if a numerical value is provided (e.g. 0), then these pixels will be set to that numerical value.

Actual behavior: If the default of None is used, the fill value is set to {}NaN{}.  This is ok for some datasets, but not the desired behavior in the pipeline, e.g. especially for datasets with no dither (WFPC2) or for drizzling only a single input frame.

Currently the MDRIZTAB parameters for ACS and WFC3 use a workaround by setting all bad pixels to good, e.g. final_bits=65535 for N=1. 

Desired behavior:   Add a new option to fill in the SCI array pixel value with either: a best guess of what it would have been OR the value from the last image in the input list. 

Users can then use the WHT image as a mask for the SCI array prior to doing photometry, if desired. 

 

stscijgbot-hstdp commented 1 year ago

Comment by Jennifer Mack on JIRA:

Note: When this work is completed, INS will deliver updated drizzle parameters.

stscijgbot-hstdp commented 1 year ago

Comment by Jennifer Mack on JIRA:

The attached figure shows the HLA product SCI (left) and WHT (right) arrays for a CRSPLIT=3 dataset (hst_05309_01_wfpc2_f336w_wf) with no dither.

For pixels with zero weight (in black), the SCI array values are not NaN and appear to be filled in with the original value.  This is the desired behavior, but the AstroDrizzle software no longer supports this.

 

 

stscijgbot-hstdp commented 1 year ago

Comment by Warren Hack on JIRA:

The way AstroDrizzle deals with WHT=0 pixels was changed under Drizzlepac PR #⁠1222.  This change completely circumvented how the original drizzle algorithm applied a fillval="INDEF", and unfortunately at the time, it was not understood what impact this would really have on WFPC2 data in particular.  Specifically, line 327 from drizzlepac/src/cdrizzlebox.c prevented AstroDrizzle from updating the ouptut image with any pixel values where WHT=0 leading to the blocks of NaN pixels in the overlap region and where bad pixels have been flagged in undithered sets of WFPC2 data.  

The difference in results with and without this line of code can be seen in the attached figures, with u65q06_img2_adrizzle_wht0_indef showing the original drizzle behavior with that line of code removed.  

The question then becomes one regarding the behavior we do want to support. 

Removing line 327 from drizzlepac/src/cdrizzlebox.c would restore the fillval="INDEF" behavior from the original drizzle algorithm.   This would, however, limit or eliminate the number of NaN pixels in the output frame. Those NaN pixels are used to define the borders of each exposure in the output frame, especially for SVM and MVM products, so tests would need to be made to confirm that pixels outside each exposure still remain as NaN where desired.  

The use of 'fillval' with floating point values gets applied using a separate function after all drizzling is completed, yet it has no way right now to allow the user to specify NaN.  This was, if I remember correctly (and I probably don't at this point), one of the motivations behind the original PR.  

The overlap between chips now gets filled in with the noisy data from the edges of the chips when fillval="INDEF" when line 327 gets removed.  This includes the regions not covered by the PC.  The logic has been developed to remove that while filling the gaps between the chips if that would make for a less confusing product.   

It is just a matter now of deciding what really should be done with these WHT=0 pixels now that we understand why the code generates the results it does now.  

stscijgbot-hstdp commented 1 year ago

Comment by Jennifer Mack on JIRA:

PR #⁠79 suggests that this change was made in Dec 2021 due to a MIRI issue with the coronographic mask.

Unfortunately, the change negatively impacts the HST drizzled products so that we can no longer fill in pixels in the active image footprint when WHT=0.   Drizzling parameters for all HST instruments were selected with the fillval=INDEF option in mind and the preferred behavior is shown in the *_wht0_indef.png, which matches the description in readthedocs.

 

P.S. This change affects ACS and WFC3, not just WFPC2.

stscijgbot-hstdp commented 9 months ago

Comment by Steve Goldman on JIRA:

Warren Hack Has there been any update for this ticket? Did folks settle on a solution, and if so, are any changes requested?

stscijgbot-hstdp commented 8 months ago

Comment by Steve Goldman on JIRA:

Jennifer Mack After discussions with Warren, Michele, and I, did we decide whether the original suggestion is still the desired outcome?

stscijgbot-hstdp commented 8 months ago

Comment by Jennifer Mack on JIRA:

Sorry for the delay in replying.  I do still think we should implement this change and I'm happy with the results for WFPC2 posted above (i'd rather leave noise near the edges than leave holes.  Specifically:

Removing line 327 from drizzlepac/src/cdrizzlebox.c would restore the fillval="INDEF" behavior from the original drizzle algorithm.   This would, however, limit or eliminate the number of NaN pixels in the output frame. Those NaN pixels are used to define the borders of each exposure in the output frame, especially for SVM and MVM products, so tests would need to be made to confirm that pixels outside each exposure still remain as NaN where desired.  

I am happy to do the testing required to confirm that pixels outside the field of view remain NaN and pixels inside the FOV are filled with their original value (with a value of zero in the WHT image).  That allows users to easily fill pixels with WHT=0 with NaN if preferred, without redrizzling.

stscijgbot-hstdp commented 8 months ago

Comment by Steve Goldman on JIRA:

Okay, thanks Jennifer Mack. So in moving ahead I see a couple things that we need to prepare for.

After the change to line 327 is made we need to:

stscijgbot-hstdp commented 7 months ago

Comment by Steve Goldman on JIRA:

After discussing this with Warren, I don't think that any added functionality is necessary. The fill_val option is that only thing needed, and that is already an available option. 

stscijgbot-hstdp commented 7 months ago

Comment by Steve Goldman on JIRA:

closed by #⁠1767

stscijgbot-hstdp commented 7 months ago

Comment by Jennifer Mack on JIRA:

Hi Steve, 

Quick question:  Will we still be able to use NaN (None) as an option for fillval?

We want to keep the 2 options currently available (NaN or integer) and then add back the 3rd option (INDEF).

Let me know if there is code you want me to test.

 

stscijgbot-hstdp commented 7 months ago

Comment by Steve Goldman on JIRA:

Hey Jenn, 

My understanding is that the fillval options should still be available. Please test the fix which is available on the current main branch of drizzlepac.

stscijgbot-hstdp commented 7 months ago

Comment by Jennifer Mack on JIRA:

Can you send me instructions on how to do that? Sorry I'm not really a software expert... just a user.

stscijgbot-hstdp commented 7 months ago

Comment by Steve Goldman on JIRA:

Ahhh, sorry of course, no worries. 

if you have git installed, the following will download drizzlepac to a local folder. 

git clone https://github.com/spacetelescope/drizzlepac.git ```

if not, you can download it by clicking on the green button at this [link](https://github.com/spacetelescope/drizzlepac)

 

Once downloaded change to the drizzlepac directory and run the following command:
```java
python setup.py develop