Open stscijgbot opened 4 years ago
Comment by Alicia Canipe: We should retest this now that Karl fixed the bug with the noise in outlier detection. Maybe it's improved?
Comment by Bryan Hilbert: I just re-ran outlier detection using version 1.2.3 of the jwst package. Using default parameter values, it is now flagging ~55,000 outliers per integration. There seems to be a strange double peak in the histogram of the number of outlier flags per integration. Many integrations are around 55,000, and one integration out of every ~6 or 7 has about 380.
Here's a list of the values for the 60 integrations in my file:
Int: 0, New hits: 55421
Int: 1, New hits: 384
Int: 2, New hits: 55428
Int: 3, New hits: 55409
Int: 4, New hits: 55401
Int: 5, New hits: 40388
Int: 6, New hits: 389
Int: 7, New hits: 54228
Int: 8, New hits: 54192
Int: 9, New hits: 55401
Int: 10, New hits: 392
Int: 11, New hits: 40421
Int: 12, New hits: 55375
Int: 13, New hits: 54204
Int: 14, New hits: 54218
Int: 15, New hits: 40404
Int: 16, New hits: 374
Int: 17, New hits: 386
Int: 18, New hits: 54188
Int: 19, New hits: 55452
Int: 20, New hits: 5851
Int: 21, New hits: 5811
Int: 22, New hits: 54195
Int: 23, New hits: 5810
Int: 24, New hits: 5828
Int: 25, New hits: 54220
Int: 26, New hits: 40453
Int: 27, New hits: 55447
Int: 28, New hits: 379
Int: 29, New hits: 381
Int: 30, New hits: 54140
Int: 31, New hits: 54404
Int: 32, New hits: 2148
Int: 33, New hits: 58534
Int: 34, New hits: 45098
Int: 35, New hits: 55547
Int: 36, New hits: 55094
Int: 37, New hits: 1505
Int: 38, New hits: 55897
Int: 39, New hits: 428
Int: 40, New hits: 55350
Int: 41, New hits: 378
Int: 42, New hits: 5860
Int: 43, New hits: 54245
Int: 44, New hits: 54216
Int: 45, New hits: 55452
Int: 46, New hits: 40397
Int: 47, New hits: 383
Int: 48, New hits: 55415
Int: 49, New hits: 40396
Int: 50, New hits: 370
Int: 51, New hits: 55442
Int: 52, New hits: 54268
Int: 53, New hits: 54188
Int: 54, New hits: 5827
Int: 55, New hits: 54221
Int: 56, New hits: 55420
Int: 57, New hits: 55393
Int: 58, New hits: 54223
Int: 59, New hits: 55472
Comment by Howard Bushouse: [~hilbert] so obviously the number of pixels being flagged is now greatly reduced using the updated algorithm for stacked TSO data, but is still flagging ~10% of the pixels. Have you experimented with any non-default parameters settings that might be a better match to the actual characteristics of NIRCam imaging data? Back in March of this year a pars-outlierdetection file was implemented in CRDS for NIRCam, but all it does is turn off the resampling option in outlier_detection when processing TSO exposures, so that the data are examined in detector space, without changing any of the other step parameters.
Comment by Bryan Hilbert: I haven't experimented with this lately. But perhaps [~eas342], [~arshn2000], or [~nnikolov] have?
Comment by Nikolay Nikolov: Are we referring to the jump detection step here? If so, I lastly ran the pipeline in April this year with result.jump.rejection_threshold = 150. to avoid any outlier detections. I haven't experimented to establish a specific value for the threshold, though.
Comment by Howard Bushouse: No, the outlier_detection step in calwebb_tso3. It examines the stack of images/spectra from each integration in a fully calibrated exposure (calints products) to look for any additional outliers that may have been missed in the calwebb_detector1 jump step.
Comment by Nikolay Nikolov: I haven't had a chance to experiment with that step too.
Issue JP-1285 was created by Bryan Hilbert:
I have a simulated NIRCam imaging TSO dataset that I have run through all stages of the pipeline. The exposure is composed of 70 integrations, and uses the GRISM256 aperture (2048 x 256 pixels).
When I run the data through the outlier detection step, I'm finding that in each integration, it is flagging 180,000 - 430,000 pixels (out of the total of 525,000 pixels). Perhaps the default thresholds being used are too strict, and it is flagging noise.
Files are located in:
/ifs/jwst/wit/witserv/data7/nrc/SSB_build_7.4_testing/TSO/imaging_TSO/data/level2b/*calints.fits
and
/ifs/jwst/wit/witserv/data7/nrc/SSB_build_7.4_testing/TSO/imaging_TSO/data/level3/*crfints.fits