Threshold TSO data by variance-computed weights in outlier detection

stscijgbot-jp commented 1 month ago

Issue JP-3779 was created on JIRA by Ned Molter:

This ticket corresponds to a comment I found in the outlier detection TSO code. It reads:

Prior to PR #⁠8473, the build_driz_weight function was used to create the weights for the input models for TSO data. However, that function was simply returning a copy of the DQ array because the var_noise was not being passed in by calwebb_tso3. As of PR #⁠8473, a cube model that includes the var_noise is passed into TSO outlier detection, so build_driz_weight would weight the cube model by the variance. Therefore build_driz_weight was removed in order to preserve the original behavior. If it is determined later that exposure time or inverse variance weighting should be used here, build_driz_weight should be re-implemented.

I have no idea whether or not the instrument teams want to implement this, i.e., if weighting based on the var_noise is desired for TSO data. However, the way the code is currently written, the maskpt parameter is useless because it sees a weight array that is all either 0 or 1. It would be helpful for clarity of both documentation and code to fix that (see this [GitHub issue|[https://github.com/spacetelescope/jwst/issues/8879]).] And it would be nice to remove the long comment.

stscijgbot-jp commented 1 month ago

Comment by David Law on JIRA:

Tagging Ian Wong ; any thoughts?

stscijgbot-jp commented 1 month ago

Comment by Ian Wong on JIRA:

For TSOs, my understanding is that outlier_detection does not resample the integrations. So in this sense, having the variance included would be to allow for filtering of the points that are included in the moving median across the integration axis and for the snr filtering. Is that correct?

stscijgbot-jp commented 1 month ago

Comment by Ned Molter on JIRA:

Of course it could theoretically work however you / the instrument teams think is best. But yes I think your comment nicely summarizes what the comment is trying to say. If we were to put build_driz_weight back in, this would compute the weights on un-flagged data based on either the inverse variance or the measurement time. Then the weights would be compared to a cutoff value, and the low-weight points would be excluded from the rolling median calculation.

Here's the source code of build_driz_weight if you are interested in that: https://github.com/spacetelescope/jwst/blob/main/jwst/resample/resample_utils.py#L182

stscijgbot-jp commented 1 month ago

Comment by David Law on JIRA:

It's unclear to me whether we necessarily would want to use var_noise weights to determine which pixels to reject from the moving median, but it seems like it would be useful to have the plumbing hooked up regardless.

One note of caution though- I believe maskpt isn't specified in param ref files (at least for MIRI), meaning that it'll revert to the code default of 0.7, meaning that we might get quite different results from this step once things are hooked up to pass weights through. We should ensure that this doesn't result in any surprises by keeping the current 0/1 weighting unless something else is specified.