Outlier detection flagging too many pixels with default settings

[stscijgbot]

Issue JP-1285 was created by Bryan Hilbert:

I have a simulated NIRCam imaging TSO dataset that I have run through all stages of the pipeline. The exposure is composed of 70 integrations, and uses the GRISM256 aperture (2048 x 256 pixels). 

When I run the data through the outlier detection step, I'm finding that in each integration, it is flagging 180,000 - 430,000 pixels (out of the total of 525,000 pixels). Perhaps the default thresholds being used are too strict, and it is flagging noise.

Files are located in:

/ifs/jwst/wit/witserv/data7/nrc/SSB_build_7.4_testing/TSO/imaging_TSO/data/level2b/*calints.fits

and

/ifs/jwst/wit/witserv/data7/nrc/SSB_build_7.4_testing/TSO/imaging_TSO/data/level3/*crfints.fits

 

[stscijgbot]

Comment by Alicia Canipe: We should retest this now that Karl fixed the bug with the noise in outlier detection. Maybe it's improved?

[stscijgbot]

Comment by Bryan Hilbert: I just re-ran outlier detection using version 1.2.3 of the jwst package. Using default parameter values, it is now flagging ~55,000 outliers per integration. There seems to be a strange double peak in the histogram of the number of outlier flags per integration. Many integrations are around 55,000, and one integration out of every ~6 or 7 has about 380. 

Here's a list of the values for the 60 integrations in my file:

Int: 0, New hits: 55421

Int: 1, New hits: 384

Int: 2, New hits: 55428

Int: 3, New hits: 55409

Int: 4, New hits: 55401

Int: 5, New hits: 40388

Int: 6, New hits: 389

Int: 7, New hits: 54228

Int: 8, New hits: 54192

Int: 9, New hits: 55401

Int: 10, New hits: 392

Int: 11, New hits: 40421

Int: 12, New hits: 55375

Int: 13, New hits: 54204

Int: 14, New hits: 54218

Int: 15, New hits: 40404

Int: 16, New hits: 374

Int: 17, New hits: 386

Int: 18, New hits: 54188

Int: 19, New hits: 55452

Int: 20, New hits: 5851

Int: 21, New hits: 5811

Int: 22, New hits: 54195

Int: 23, New hits: 5810

Int: 24, New hits: 5828

Int: 25, New hits: 54220

Int: 26, New hits: 40453

Int: 27, New hits: 55447

Int: 28, New hits: 379

Int: 29, New hits: 381

Int: 30, New hits: 54140

Int: 31, New hits: 54404

Int: 32, New hits: 2148

Int: 33, New hits: 58534

Int: 34, New hits: 45098

Int: 35, New hits: 55547

Int: 36, New hits: 55094

Int: 37, New hits: 1505

Int: 38, New hits: 55897

Int: 39, New hits: 428

Int: 40, New hits: 55350

Int: 41, New hits: 378

Int: 42, New hits: 5860

Int: 43, New hits: 54245

Int: 44, New hits: 54216

Int: 45, New hits: 55452

Int: 46, New hits: 40397

Int: 47, New hits: 383

Int: 48, New hits: 55415

Int: 49, New hits: 40396

Int: 50, New hits: 370

Int: 51, New hits: 55442

Int: 52, New hits: 54268

Int: 53, New hits: 54188

Int: 54, New hits: 5827

Int: 55, New hits: 54221

Int: 56, New hits: 55420

Int: 57, New hits: 55393

Int: 58, New hits: 54223

Int: 59, New hits: 55472

[stscijgbot]

Comment by Howard Bushouse: [~hilbert] so obviously the number of pixels being flagged is now greatly reduced using the updated algorithm for stacked TSO data, but is still flagging ~10% of the pixels. Have you experimented with any non-default parameters settings that might be a better match to the actual characteristics of NIRCam imaging data? Back in March of this year a pars-outlierdetection file was implemented in CRDS for NIRCam, but all it does is turn off the resampling option in outlier_detection when processing TSO exposures, so that the data are examined in detector space, without changing any of the other step parameters.

[stscijgbot]

Comment by Bryan Hilbert: I haven't experimented with this lately. But perhaps [~eas342], [~arshn2000],  or [~nnikolov] have?

[stscijgbot]

Comment by Nikolay Nikolov: Are we referring to the jump detection step here? If so, I lastly ran the pipeline in April this year with result.jump.rejection_threshold = 150. to avoid any outlier detections. I haven't experimented to establish a specific value for the threshold, though.  

[stscijgbot]

Comment by Howard Bushouse: No, the outlier_detection step in calwebb_tso3. It examines the stack of images/spectra from each integration in a fully calibrated exposure (calints products) to look for any additional outliers that may have been missed in the calwebb_detector1 jump step.

[stscijgbot]

Comment by Nikolay Nikolov: I haven't had a chance to experiment with that step too.