search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
565 stars 167 forks source link

Bad reference file slice indexes #2469

Closed stscieisenhamer closed 5 years ago

stscieisenhamer commented 6 years ago

Issue

From EMTB DGA#3, calwebb_detector1 on a NIRISS with exptype == "nis_extcal" gives the following:

Log output

``` strun cfgs/calwebb_detector1.cfg jw00610002001_02101_00001_nis_uncal.fits 2018-09-06 13:31:32,767 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/core.py:71: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. def __init__(self, sigma_clip=SigmaClip(sigma=3., iters=5)): 2018-09-06 13:31:32,767 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/core.py:114: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. def __init__(self, sigma_clip=SigmaClip(sigma=3., iters=5)): 2018-09-06 13:31:32,770 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/background_2d.py:285: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. sigma_clip=SigmaClip(sigma=3., iters=10), 2018-09-06 13:31:33,197 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created. 2018-09-06 13:31:33,198 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created. 2018-09-06 13:31:33,199 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created. 2018-09-06 13:31:33,200 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created. 2018-09-06 13:31:33,200 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created. 2018-09-06 13:31:33,201 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created. 2018-09-06 13:31:33,202 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created. 2018-09-06 13:31:33,203 - stpipe.Detector1Pipeline.rscd - INFO - RSCD_Step instance created. 2018-09-06 13:31:33,204 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created. 2018-09-06 13:31:33,205 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created. 2018-09-06 13:31:33,205 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created. 2018-09-06 13:31:33,206 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created. 2018-09-06 13:31:33,207 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created. 2018-09-06 13:31:33,208 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created. 2018-09-06 13:31:33,209 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created. 2018-09-06 13:31:33,210 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created. 2018-09-06 13:31:33,211 - stpipe - INFO - Hostname: inanna2.stsci.edu 2018-09-06 13:31:33,211 - stpipe - INFO - OS: Darwin 2018-09-06 13:31:33,247 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('jw00610002001_02101_00001_nis_uncal.fits',). 2018-09-06 13:31:33,563 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'jw00610002001_02101_00001_nis_uncal.fits' reftypes = ['dark', 'gain', 'linearity', 'mask', 'persat', 'readnoise', 'refpix', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars'] CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_wavelengthrange_0008.rmap 945 bytes (1 / 3 files) (0 / 3.5 K bytes) 2018-09-06 13:31:34,980 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_wavelengthrange_0008.rmap 945 bytes (1 / 3 files) (0 / 3.5 K bytes) CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_0107.imap 2.0 K bytes (2 / 3 files) (945 / 3.5 K bytes) 2018-09-06 13:31:35,125 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_0107.imap 2.0 K bytes (2 / 3 files) (945 / 3.5 K bytes) CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_0482.pmap 573 bytes (3 / 3 files) (2.9 K / 3.5 K bytes) 2018-09-06 13:31:35,178 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_0482.pmap 573 bytes (3 / 3 files) (2.9 K / 3.5 K bytes) 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_dark_0116.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_gain_0005.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_linearity_0011.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_mask_0010.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_persat_0003.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for READNOISE reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_readnoise_0001.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for REFPIX reference file is 'N/A'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is 'N/A'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_saturation_0010.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_superbias_0122.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_trapdensity_0002.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_trappars_0002.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ... 2018-09-06 13:31:35,571 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (,). 2018-09-06 13:31:35,588 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES and FRMDIVSR are equal; correction not needed 2018-09-06 13:31:35,588 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped 2018-09-06 13:31:35,589 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done 2018-09-06 13:31:35,633 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (,). 2018-09-06 13:31:35,654 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_mask_0010.fits 2018-09-06 13:31:37,313 - stpipe.Detector1Pipeline.dq_init - WARNING - Keyword CDP_WARM does not correspond to an existing DQ mnemonic, so will be ignored 2018-09-06 13:31:37,313 - stpipe.Detector1Pipeline.dq_init - WARNING - Keyword CDP_NOISY does not correspond to an existing DQ mnemonic, so will be ignored 2018-09-06 13:31:37,368 - stpipe.Detector1Pipeline.dq_init - INFO - Extracting mask subarray to match science data 2018-09-06 13:31:37,378 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done 2018-09-06 13:31:37,427 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (,). 2018-09-06 13:31:37,448 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_saturation_0010.fits 2018-09-06 13:31:37,690 - stpipe.Detector1Pipeline.saturation - INFO - Extracting reference file subarray to match science data 2018-09-06 13:31:37,702 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done 2018-09-06 13:31:37,750 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (,). 2018-09-06 13:31:37,750 - stpipe.Detector1Pipeline.ipc - INFO - Step skipped. 2018-09-06 13:31:37,751 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done 2018-09-06 13:31:37,793 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias running with args (,). 2018-09-06 13:31:37,811 - stpipe.Detector1Pipeline.superbias - INFO - Using SUPERBIAS reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_superbias_0122.fits 2018-09-06 13:31:37,865 - stpipe.Detector1Pipeline.superbias - ERROR - Computed reference file slice indexes are incompatible with size of reference data array 2018-09-06 13:31:37,865 - stpipe.Detector1Pipeline.superbias - ERROR - xstart=797, xstop=877, ystart=1968, ystop=2048 2018-09-06 13:31:37,866 - stpipe.Detector1Pipeline.superbias - ERROR - Reference xsize=80, ysize=80 ---------------------------------------------------------------------- ERROR RUNNING STEP 'Detector1Pipeline': Bad reference file slice indexes Traceback (most recent call last): File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline step.run(*positional) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_detector1.py", line 106, in process input = self.superbias(input) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/superbias_step.py", line 41, in process result = bias_sub.do_correction(input_model, bias_model) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/bias_sub.py", line 37, in do_correction bias_model = reffile_utils.get_subarray_model(input_model, bias_model) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/lib/reffile_utils.py", line 344, in get_subarray_model raise ValueError('Bad reference file slice indexes') ValueError: Bad reference file slice indexes ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/EGG-INFO/scripts/strun", line 14, in step = Step.from_cmdline(sys.argv[1:]) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 162, in from_cmdline return cmdline.step_from_cmdline(args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline step.run(*positional) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_detector1.py", line 106, in process input = self.superbias(input) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/superbias_step.py", line 41, in process result = bias_sub.do_correction(input_model, bias_model) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/bias_sub.py", line 37, in do_correction bias_model = reffile_utils.get_subarray_model(input_model, bias_model) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/lib/reffile_utils.py", line 344, in get_subarray_model raise ValueError('Bad reference file slice indexes') ValueError: Bad reference file slice indexes ```

stscieisenhamer commented 6 years ago

Resources

hbushouse commented 6 years ago

Theoretically none of these should be an issue within CAL. The errors arise when CAL is given SUBSTRT1/2 and data array size values that result in it trying to extract a subarray from a full-frame reference file using slice coords that are outside of the reference data array. So these are actually errors in the SUBSTRT1/2 values that are populated in level-1b products by SDP. There are many CV3 data sets that have been known for a long time to have this issue, usually due to missing or bad EngDB coverage for the exposures (that's where SDP gets the subarray corner values).

hbushouse commented 6 years ago

jw00612006001_02101_00001_nrs1_uncal.fits is a case of no coverage within CRDS for readnoise ref files matching the exposure. Readnoise refs are selected on DETECTOR, READPATT, and SUBARRAY. The exposure has DETECTOR='NRS1', READPATT='NRSIRS2', and SUBARRAY='FULL'. There's no readnoise ref file for READPATT='NRSIRS2' (only 'NRSIRS2RAPID'), so it defaults to the one reffile that has READPATT='N/A', which is jwst_nirspec_readnoise_00014.fits, but that is a 2048x256 subarray, which obviously can't be applied to a full-frame science exposure.

stscicrawford commented 6 years ago

This should probably be a ticket open in the Jira CRDS project with a request for how to handle it and/or for a new reference file to be submitted.

hbushouse commented 6 years ago

jw00610002001_02101_00001_nis_uncal.fits is well-known case of the NIRISS SUB80 having SUBSTRT1/2 values populated by SDP that are off by 1.

jw80700012001_02102_00001_mirimage_uncal.fits is a SUBARRAY='BRIGHTSKY' exposure, where the science exposure has SUBSTRT1=115, but the matching reffile jwst_miri_readnoise_0048.fits has SUBSTRT1=457, so the pipeline tries to extract a different portion of the reffile (which fails).

hbushouse commented 6 years ago

jw87600029001_02101_00001_nis_uncal.fits is a previously-known case of the science exposure having SUBARRAY='SUB80' with SUBSTRT1=1005, but the reffile jwst_niriss_superbias_0122.fits has SUBSTRT1=1045, because the subarray location got moved since the CV3 data were acquired.

hbushouse commented 6 years ago

https://jira.stsci.edu/browse/CRDS-237 has been filed for the NIRSpec missing readnoise coverage.

hbushouse commented 6 years ago

@stscieisenhamer Can we confirm whether all of the datasets that resulted in this error with SUBARRAY='FULL' are NIRSpec with READPATT='NRSIRS2' (and hence a result of the hole in CRDS coverage)?

All the others that are subarray exposures must be due to mismatches in SUBSTRT1/2 keyword values and aren't our problem to fix.

hbushouse commented 5 years ago

All of these issues are problems in SDP (bad subarray corner values) or CRDS (missing or incorrect coverage). So I'm closing this for CAL.