Saturated Pixel Propagation

[stscijgbot-jp]

Issue JP-1842 was created on JIRA by Marcio Melendez Hernandez:

When there is saturation in all the groups inside an integration per pixel, the pipeline assigns a default value of zero counts because the pipeline doesn’t have any good groups to work with. However, the Level 3 pipeline (i2d products) is not propagating the zero value into its corresponding coordinate. Because of this, the core of a saturated PSF has pixels with zero counts in the individual dither exposure BUT the L3 products has non-zero values in the core. I think the default value of zero, for a saturated pixel (coordinate), should propagate directly to a zero count in the L3 product otherwise we are assigning non-realistic values to the core of the PSFs. In other words, a saturated PSF in the individual dithers exposures (L2b) should remain saturated in the L3 product. 

Note that because this is happening in the calwebb_image3, the same behavior could be affecting other instruments as well. 

 

Example: 

For a particular program, OTE-28, we are looking for small changes in the core of the PSF. Thus, if a saturated PSF (in the Level 2b, _cal.fits) shows an unrealistic  non-saturated core in the i2d product, it could be a potential problem for some of our analyses. The script to create the association files and to run the pipeline are at: 

/ifs/jwst/tel/TeamPractices/run_jwst_level3.py

Results from our simulations are under:

/ifs/jwst/tel/wfr4_mirage_sims/psf_check/OTE-28_New_Catalog_Ball_PSFs/outputs/outputs/nominal/

 

!Screen Shot 2020-12-23 at 10.50.25 AM.png|thumbnail!

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I initially misunderstood the issue here.  It sounded like the proposal was to have zero values in any individual exposure get mapped to zero values in the combined i2d image.  That I think we do not want to be the case.  But with only a single image being drizzled to the output image I suppose the pixels in the resampled image that only partially overlap a pixel with a zero value in the raw image get a non-zero value because some part of the signal is coming from pixels that have signal in the original image and some part is coming from the saturated pixel.  I am not too conversant with what drizzle does, but for NIRISS when I look at some simulated images with saturated star cores I do not see something like the example above (where there are say 7 saturated pixels in the calibrated image and only 1 such in the resampled image).  It does look like a similar effect is present but it is not as "aggressive" in putting signal into the saturated pixels in the core.  Hence the effect might be sensitive to some of the drizzle parameters being used.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Theoretically, if all groups in all integrations for a given pixel are flagged as saturated, the ramp_fit step not only sets the output slope to zero, but also propagates the SATURATED flag and sets the DO_NOT_USE flag for that pixel. Assuming all DQ propagation happens correctly throughout all the remaining level-2b (calwebb_image2) steps, the "cal" products should have the DO_NOT_USE flag set for those pixels. That should then cause the resample (drizzle) step to not include any contribution from that pixel when drizzling flux into the resampled image. But regardless of whether you're drizzling a single image or a set of dithered images, bright sources will likely have their cores saturated and flagged as DO_NOT_USE in all of the inputs and hence there's no way to recover useful flux values for the core in the resampled image, and I agree that in such cases the unrecoverable pixels should have DQ flags set in the resampled image to indicate that these pixels are not useable, even if they happened to end up with a non-zero flux value. Is that the case in the example data mentioned above (are DQ flags set in the drizzled image)? Could you provide a pointer to the location of the example data files?

There could be cases where the core of an unresolved point source saturates only when centered in a given pixel and hence if dithering is applied and the core gets shifted to straddle two or more pixels, it may not completely saturate in those images, and hence some of the core signal could be recovered when combining multiple images.

[stscijgbot-jp]

Comment by James Davies [X] on JIRA:

Agree Howard.

Yes, DO_NOT_USE should be propagated through the _rate and _cal products and those pixels not used in resampling. Since this ticket was filed with version 0.17.1, it's possible this was broken then and has been fixed subsequently. This can be checked, but I don't think that is the issue here.

That is because if there are no good groups, and thus no rate can be produced for a particular pixel, then that value should be flagged as DO_NOT_USE and the rate should be set to zero. If the pixel is zero in the _rate file, there's no way for anything downstream in the _cal file, including resample, to be using anything but zero, even if it is not respecting DO_NOT_USE.

So the thing to do is check whether these pixels that you don't think should have flux are getting flux from adjacent pixels. I.e. are other dithers in the association contributing to the output flux in the _i2d image. The way to do this is to check the WHT and CON images.

A clarification: there are no DQ flags in resampled products. Only weight and context (WHT and CON). And both are important in helping you figure out where your signal is coming from in each pixel of the _i2d.

Finally, if you're interested in small changes in the cores of bright sources, probably resampled images that have been drizzled introduce lots of complexities to this analysis. I would tread very carefully. =)

A final note, that script pointed to above /ifs/jwst/tel/TeamPractices/run_jwst_level3.py uses Image3Pipeline.run instead of Image3Pipeline.call, which means it won't pick up any parameter reference files for either the pipeline or the steps within the pipeline. Something to be aware of.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

A quick look at the example images provided above confirms that the pixels in the cores of the saturated stars that have SCI=0 in the cal product have DQ flag values with odd numbers (3, 7) indicating that saturation and/or jumps were detected (2, 4) and that the DO_NOT_USE flag is set. In the i2d image CON (context) array, pixels outside of the saturated core have values of 15, indicating that all inputs contributed to the resampled values, while the saturated pixels near the resampled core drop to CON values of 11, 6, 5, and finally zero, indicating that some, but not all images had contributions to the resampled pixels, due to picking up some fraction of the flux from neighboring pixels that were not flagged with DO_NOT_USE. Similarly, the WHT array shows the same behavior, with good pixels outside the core having values of 150-215, then dropping to values of 110, 90, 32, 15, 8, and finally 0 for the half dozen or so pixels in the saturated regime.

As James Davies [X] pointed out in the previous comment, there are no DQ arrays in resampled images (because it's semi-impossible to redistribute contributions of flags in the same way you redistribute flux), so any analysis routines should make use of the WHT array values, which in this case would drastically reduce the contribution level from the pixels in and near the saturated cores.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Given that we've verified that the various pipeline steps seem to be working as designed:

1) pixels that have no unsaturated groups get zeroed out in unrectified images 2) some flux from neighboring pixels gets redistributed into those locations when dithered images are combined and rectified 3) the CON and WHT arrays of the combined/rectified images properly reflect the fact that these pixels carry less or no weight

I don't know what else can be done, at least within the pipeline steps. Analysis tools will need to use the WHT and/or CON array values to know which pixels to use or how much to trust their values.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Without further direction from the Cal WG or DMS WG, there's nothing that Cal can do to alleviate this issue, because - as reported above - all of the Cal code seems to be doing what it was designed to do. So at this point I think we need to throw the ball back into the INS court for discussion as what, if anything, should be done to any of the Cal algorithms (Alicia Canipe Anton Koekemoer)

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Thanks Howard Bushouse!

Marcio Melendez Hernandez, what are your thoughts based on the input from Howard and James? Let us know what you'd like to do with this ticket. If the algorithm needs to be tweaked, we'd have to take the discussion back to Anton Koekemoer and the CalWG. 

[stscijgbot-jp]

Comment by Marcio Melendez Hernandez on JIRA:

Using WHT and/or CON to find the level of uncertainty in some of these pixel values make sense and it is in agreement with our current plan on how to deal with this issue. 

Thanks! 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Okay, thank you Marcio Melendez Hernandez!

So Howard Bushouse, think we can go ahead and close this ticket to avoid confusion based on how Marcio's team will proceed? Then, if we need to open a new one to tweak the algorithm or fix another bug we can always do that. 

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

I think we should set all level 2 pixels that have DO NOT USE set to have a SCI value of NaN. It is misleading to have a valid SCI value for a pixel that is invalid. Users who use the level 2 products for further processing will be misled. 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

When you say "level 2" products are you referring only to unrectified products, e.g. rate and cal, or also the single-image resampled "i2d" products? We've already established that the reason for some small amount of signal contained in the resampled versions of images is due to flux from neighboring pixels getting included. The unresampled products, meanwhile, current have SCI=0. Are you suggesting we simply change this to be NaN (in rate and cal products) instead of zero? Just to make it more obvious that the pixel is bad?

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

The DO_NOT_USE pixels are readily identified by the zero signal values.  Personally I do not like setting NaN values in data arrays and do not support the idea of setting to NaN when the pixels already are easy to identify and masked out as needed.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

My concern is that zeros can just pass through like they are normal values. I know that NaNs are annoying but that's their purpose. The slope is "Not a Number". If you are looking for them, they are easy to deal with in both IDL and Python.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

This was discussed in the calibration working group at some point and it was decided at that time to set any such pixels to a signal of 0.0.  But the drizzle step does initially have access to the DQ flags of the input _cal.fits images and it should disregard DO_NOT_USE pixels even if these DQ flags are not propagated down to level3 products.  Is that what is happening, or not?  If it is what is happening, then I do not understand what the issue is for Mike, since setting the DO_NOT_USE pixels to NaN in level2 will not affect the drizzle outputs in level3 or level 2 in such a case as far as I know.  What happens downstream from that in the _i2d.fits where the DQ values are not available is a different matter.  When the final level 3 resampled product is produced does the step work with the individual _cal.fits files from level 2 or does it work with the individual _i2d.fits files?  I thought it was the former not the latter.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Kevin Volk level 3 image combination does indeed work with the individual level-2b cal images as input, which in this case have completely saturated pixels set to zero and DQ=DO_NOT_USE. Upon input to resample, those pixels with DO_NOT_USE are in fact disregarded. But in these cases, where there are neighboring pixels with OK flux values, a little bit of that neighboring good flux gets drizzled into the output pixel that was predominantly associated with the original input pixel that had no good flux. So the resampled image no longer has SCI=0, but it does have very low WHT and CON values to indicate that only a very small fraction of good input pixels contributed to the output.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

So given this, it seems that if we set the level 2 products to have NaN where there is no ramp data the output drizzled products will not be any different.  If one pixel in the core saturates in say all four input images in a set then the level 3 resampled image will have this small signal in the central pixel of the core just as currently.  Hence if this aspect of the resampled images is considered a problem then the problem does not change whether we have the level 2 pixels with no good ramp data set to 0.0 or set to NaN.  If all this is correct, I do not see that setting the DO_NOT_USE pixels to NaN would improve anything.  Anyone looking at the individual _cal images must know that a signal of zero with an uncertainty of zero is not a real measurement.  If people go straight to the resampled images there is no immediate indication in the images that there might be an issue with the resamped core pixel.  They would have to look at the WHT and CON values.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Given that negative slopes are valid values, it is not true that 0.0 is an invalid value. I agree that users inspect the slopes of their level 2 products by eye they will see that a 0.0 value in the center of a star is probably wrong. BUT if they don't and just run things through their own pipeline they will be misled. Just like the WF shadow folks were when they first saw zero values in the cores of stars. 

I still don't understand why you want to put a 0.0 slope when we don't know the slope. I'd prefer -10^10 to zero. Zero is in the valid output range while -10^10 is not a valid slope. 

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I have no objection to changing the marker to say -1.e+10 if that seems better.  A slope of 0.0 is not valid for real data, just as negative slopes are not valid for real data in the sense that there is no such thing as a negative signal on the sky, nor zero either.  What I want to avoid are things that cannot be displayed as numbers either in an image display or when doing math.

 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Marcio Melendez Hernandez  and others, for opening this ticket and for the discussion so far. One item that would be useful would be if you could make available please the level2 i2d.fits files for each of the individual exposures (right now I find only the combined level3 output jw01165001_g1i2d.fits in the directory, and not the separate level2 jw01165001001*_i2d.fits files that I'd have expected to have been produced by calwebb_image2 - can you let me know please where I can find those?

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Scheduled for discussion in JWST Cal WG Meeting 2022-03-01

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The conclusion at the end of Cal WG meeting was:

So, outcome from today's discussion in the broader context of pipeline i2d products:

- pipeline i2d products currently have 0 as the default output value when there are no valid input pixels
- for those who want something different, eg NaN, this can be readily achieved by rerunning the pipeline offline and setting fillval=NaN
- The question of whether to change the default value of fillval in the operational pipeiine, to populate such pixels not with 0 but with NaN, should be revisited with broader input from CalWG reps - so the team reps are asked to consider the implications either way (0 vs NaN) for revisiting this in a future discussion.```
Hence there is still no indication of any desired/needed changes to the Cal code, at least until a decision is made in the future to change the default value of the "fillval" parameter in the resample step. Given that this can and most likely should be accomplished via updated settings in either the "drizpars" reference file or the pars-resamplestep reference file for each instrument and mode, rather than in the resample step code itself, I don't see any potential Cal code changes for this. 

So can this ticket either be closed or reassigned to instrument teams to deliver updated ref files?

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

This issue remained a topic of discussion, and was brought to conclusion at JWST Cal WG Meeting 2022-11-01

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

JP-2988 and JP-3014 have now been implemented in DMS B9.1, which include changes to the propagation of saturation flags (only set in downstream products if pixel is totally saturated) and setting no-slope pixels to NaN instead of zero.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Anton Koekemoer Is anyone in INS interested in testing this or should we close the ticket?

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thanks Howard, I'll tag Marcio Melendez Hernandez  for testing it, who was the original reporter on this ticket, and we can hear back about whether this is now working as intended.

[stscijgbot-jp]

Comment by Marcio Melendez Hernandez on JIRA:

Anton Koekemoer My apologies, I missed the latest updates to this ticket. I can test this if still needed. Thanks!

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Marcio Melendez Hernandez , yes, it would be great if you can confirm that the fix is working as intended. Thanks. Just comment in this ticket with your testing results.

[stscijgbot-jp]

Comment by Marcio Melendez Hernandez on JIRA:

I performed the test by running the pipeline on the uncal.fits all the way to Level 3 for DMS > B9.1. I used program 1464 that contains a 4-point dither Guider observations in order to see the effect of the saturated pixel propagation from the individual dithers to the final Level 3. 

Observations and scripts are under:

/ifs/jwst/tel/flight/OTE-28.2

 1. The saturated pixels are marked as NaN

2. The Level 3 is not as "aggressive" as before in terms of putting signal into the NaN pixels, just some pixels on the edge as there may be some flux on the partially overlapping pixels because of the dithers. So, I think this is expected. 

In addition, I also looked at some other products in MAST that have been automatically re-processed with the latest DMS built and I found the same results for the saturated pixel propagation between individual dither in the cal to the combined L3 product.   

 

Conclusion: It seems the fix is working as intended.

 

Thanks! 

 

 

!Screen Shot 2023-04-24 at 3.07.58 PM.png!