stscijgbot-jp
commented
2 years ago Comment by Rachel Cooper on JIRA:
This is also an issue for the more general Image3 outlier detection step. In particular, the documentation does not explain how the two values for SNR are used. Something along the lines of what M. Regan communicated in the Slack discussion would be helpful: "The user must specify two cut-off signal-to-noise (SNR) values for determining whether a pixel should be masked: the first for detecting the primary cosmic ray, and the second for masking lower-level bad pixels adjacent to those found in the first pass. Since cosmic rays often extend across several pixels, the adjacent pixels make use of a slightly lower SNR threshold."
Issue JP-1647 was created on JIRA by Nestor Espinoza:
The ReadTheDocs documentation for the
outlier_detectionstep in general (but for TSOs in particular) needs to have a detailed and/or clear explanation of how outliers are being detected and flagged in the pipeline.In general, the step indicates in the documentation that a "statistical model" is used to perform outlier rejection, but that model is not defined. A user might naively think that the rejection is made through a n-sigma rejection of some sort, but looking at the actual code, this is not true for all the modes (e.g., derivatives are used in the images to find thresholds --- see, e.g., flag_cr function here https://github.com/spacetelescope/jwst/blob/master/jwst/outlier_detection/outlier_detection.py). For TSOs in particular (https://jwst-pipeline.readthedocs.io/en/latest/jwst/outlier_detection/outlier_detection.html#outlier-detection-imaging), the text reads "(...) TSO data’s value, however, comes from looking for variations from one integration to the next. (...)", but currently, at least in that text it is not obvious how this integration-by-integration variation is being used for a better outlier rejection algorithm (other than not resampling). The information there provided is also inconsistent with what the pipeline apparently is doing for Imaging TSOs which, unless I'm grossly misunderstanding the code (do_detection function in https://github.com/spacetelescope/jwst/blob/master/jwst/outlier_detection/outlier_detection_scaled.py), the median image is apparently being re-scaled according to the flux in an aperture on each image. This rescaling is apparently not done for SpectroscopicTSOs (which might be an issue, but I'll leave that for another possible ticket).
Marking this as Major because this might prevent understanding of why the pipeline is flagging outliers at all (it has been my case for a couple of days now), but feel free to lower its priority.