Closed stscieisenhamer closed 5 years ago
/grp/jwst/ssb/eisenhamer/jwst_issues/issue2469
jw00612006001_02101_00001_nrs1_uncal.fits
calwebb_guider
on jw00600004001_gs-acq1_2018221133141_uncal.fits
in same folder.jw80700012001_02102_00001_mirimage_uncal.fits
jw87600029001_02101_00001_nis_uncal.fits
Theoretically none of these should be an issue within CAL. The errors arise when CAL is given SUBSTRT1/2 and data array size values that result in it trying to extract a subarray from a full-frame reference file using slice coords that are outside of the reference data array. So these are actually errors in the SUBSTRT1/2 values that are populated in level-1b products by SDP. There are many CV3 data sets that have been known for a long time to have this issue, usually due to missing or bad EngDB coverage for the exposures (that's where SDP gets the subarray corner values).
jw00612006001_02101_00001_nrs1_uncal.fits is a case of no coverage within CRDS for readnoise ref files matching the exposure. Readnoise refs are selected on DETECTOR, READPATT, and SUBARRAY. The exposure has DETECTOR='NRS1', READPATT='NRSIRS2', and SUBARRAY='FULL'. There's no readnoise ref file for READPATT='NRSIRS2' (only 'NRSIRS2RAPID'), so it defaults to the one reffile that has READPATT='N/A', which is jwst_nirspec_readnoise_00014.fits, but that is a 2048x256 subarray, which obviously can't be applied to a full-frame science exposure.
This should probably be a ticket open in the Jira CRDS project with a request for how to handle it and/or for a new reference file to be submitted.
jw00610002001_02101_00001_nis_uncal.fits is well-known case of the NIRISS SUB80 having SUBSTRT1/2 values populated by SDP that are off by 1.
jw80700012001_02102_00001_mirimage_uncal.fits is a SUBARRAY='BRIGHTSKY' exposure, where the science exposure has SUBSTRT1=115, but the matching reffile jwst_miri_readnoise_0048.fits has SUBSTRT1=457, so the pipeline tries to extract a different portion of the reffile (which fails).
jw87600029001_02101_00001_nis_uncal.fits is a previously-known case of the science exposure having SUBARRAY='SUB80' with SUBSTRT1=1005, but the reffile jwst_niriss_superbias_0122.fits has SUBSTRT1=1045, because the subarray location got moved since the CV3 data were acquired.
https://jira.stsci.edu/browse/CRDS-237 has been filed for the NIRSpec missing readnoise coverage.
@stscieisenhamer Can we confirm whether all of the datasets that resulted in this error with SUBARRAY='FULL' are NIRSpec with READPATT='NRSIRS2' (and hence a result of the hole in CRDS coverage)?
All the others that are subarray exposures must be due to mismatches in SUBSTRT1/2 keyword values and aren't our problem to fix.
All of these issues are problems in SDP (bad subarray corner values) or CRDS (missing or incorrect coverage). So I'm closing this for CAL.
Issue
From EMTB DGA#3,
calwebb_detector1
on a NIRISS withexptype == "nis_extcal"
gives the following:Log output
``` strun cfgs/calwebb_detector1.cfg jw00610002001_02101_00001_nis_uncal.fits 2018-09-06 13:31:32,767 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/core.py:71: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. def __init__(self, sigma_clip=SigmaClip(sigma=3., iters=5)): 2018-09-06 13:31:32,767 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/core.py:114: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. def __init__(self, sigma_clip=SigmaClip(sigma=3., iters=5)): 2018-09-06 13:31:32,770 - stpipe - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/background/background_2d.py:285: AstropyDeprecationWarning: "iters" was deprecated in version 3.1 and will be removed in a future version. Use argument "maxiters" instead. sigma_clip=SigmaClip(sigma=3., iters=10), 2018-09-06 13:31:33,197 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created. 2018-09-06 13:31:33,198 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created. 2018-09-06 13:31:33,199 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created. 2018-09-06 13:31:33,200 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created. 2018-09-06 13:31:33,200 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created. 2018-09-06 13:31:33,201 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created. 2018-09-06 13:31:33,202 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created. 2018-09-06 13:31:33,203 - stpipe.Detector1Pipeline.rscd - INFO - RSCD_Step instance created. 2018-09-06 13:31:33,204 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created. 2018-09-06 13:31:33,205 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created. 2018-09-06 13:31:33,205 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created. 2018-09-06 13:31:33,206 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created. 2018-09-06 13:31:33,207 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created. 2018-09-06 13:31:33,208 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created. 2018-09-06 13:31:33,209 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created. 2018-09-06 13:31:33,210 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created. 2018-09-06 13:31:33,211 - stpipe - INFO - Hostname: inanna2.stsci.edu 2018-09-06 13:31:33,211 - stpipe - INFO - OS: Darwin 2018-09-06 13:31:33,247 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('jw00610002001_02101_00001_nis_uncal.fits',). 2018-09-06 13:31:33,563 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'jw00610002001_02101_00001_nis_uncal.fits' reftypes = ['dark', 'gain', 'linearity', 'mask', 'persat', 'readnoise', 'refpix', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars'] CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_wavelengthrange_0008.rmap 945 bytes (1 / 3 files) (0 / 3.5 K bytes) 2018-09-06 13:31:34,980 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_wavelengthrange_0008.rmap 945 bytes (1 / 3 files) (0 / 3.5 K bytes) CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_0107.imap 2.0 K bytes (2 / 3 files) (945 / 3.5 K bytes) 2018-09-06 13:31:35,125 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_nircam_0107.imap 2.0 K bytes (2 / 3 files) (945 / 3.5 K bytes) CRDS - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_0482.pmap 573 bytes (3 / 3 files) (2.9 K / 3.5 K bytes) 2018-09-06 13:31:35,178 - stpipe.Detector1Pipeline - INFO - Fetching /Users/eisenham/Downloads/crds-cache/mappings/jwst/jwst_0482.pmap 573 bytes (3 / 3 files) (2.9 K / 3.5 K bytes) 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_dark_0116.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_gain_0005.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_linearity_0011.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_mask_0010.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_persat_0003.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for READNOISE reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_readnoise_0001.fits'. 2018-09-06 13:31:35,394 - stpipe.Detector1Pipeline - INFO - Prefetch for REFPIX reference file is 'N/A'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is 'N/A'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_saturation_0010.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_superbias_0122.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_trapdensity_0002.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_trappars_0002.fits'. 2018-09-06 13:31:35,395 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ... 2018-09-06 13:31:35,571 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (,).
2018-09-06 13:31:35,588 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES and FRMDIVSR are equal; correction not needed
2018-09-06 13:31:35,588 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2018-09-06 13:31:35,589 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2018-09-06 13:31:35,633 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (,).
2018-09-06 13:31:35,654 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_mask_0010.fits
2018-09-06 13:31:37,313 - stpipe.Detector1Pipeline.dq_init - WARNING - Keyword CDP_WARM does not correspond to an existing DQ mnemonic, so will be ignored
2018-09-06 13:31:37,313 - stpipe.Detector1Pipeline.dq_init - WARNING - Keyword CDP_NOISY does not correspond to an existing DQ mnemonic, so will be ignored
2018-09-06 13:31:37,368 - stpipe.Detector1Pipeline.dq_init - INFO - Extracting mask subarray to match science data
2018-09-06 13:31:37,378 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2018-09-06 13:31:37,427 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (,).
2018-09-06 13:31:37,448 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_saturation_0010.fits
2018-09-06 13:31:37,690 - stpipe.Detector1Pipeline.saturation - INFO - Extracting reference file subarray to match science data
2018-09-06 13:31:37,702 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2018-09-06 13:31:37,750 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (,).
2018-09-06 13:31:37,750 - stpipe.Detector1Pipeline.ipc - INFO - Step skipped.
2018-09-06 13:31:37,751 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2018-09-06 13:31:37,793 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias running with args (,).
2018-09-06 13:31:37,811 - stpipe.Detector1Pipeline.superbias - INFO - Using SUPERBIAS reference file /Users/eisenham/Downloads/crds-cache/references/jwst/niriss/jwst_niriss_superbias_0122.fits
2018-09-06 13:31:37,865 - stpipe.Detector1Pipeline.superbias - ERROR - Computed reference file slice indexes are incompatible with size of reference data array
2018-09-06 13:31:37,865 - stpipe.Detector1Pipeline.superbias - ERROR - xstart=797, xstop=877, ystart=1968, ystop=2048
2018-09-06 13:31:37,866 - stpipe.Detector1Pipeline.superbias - ERROR - Reference xsize=80, ysize=80
----------------------------------------------------------------------
ERROR RUNNING STEP 'Detector1Pipeline':
Bad reference file slice indexes
Traceback (most recent call last):
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline
step.run(*positional)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_detector1.py", line 106, in process
input = self.superbias(input)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/superbias_step.py", line 41, in process
result = bias_sub.do_correction(input_model, bias_model)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/bias_sub.py", line 37, in do_correction
bias_model = reffile_utils.get_subarray_model(input_model, bias_model)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/lib/reffile_utils.py", line 344, in get_subarray_model
raise ValueError('Bad reference file slice indexes')
ValueError: Bad reference file slice indexes
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/EGG-INFO/scripts/strun", line 14, in
step = Step.from_cmdline(sys.argv[1:])
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 162, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline
step.run(*positional)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_detector1.py", line 106, in process
input = self.superbias(input)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/superbias_step.py", line 41, in process
result = bias_sub.do_correction(input_model, bias_model)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/superbias/bias_sub.py", line 37, in do_correction
bias_model = reffile_utils.get_subarray_model(input_model, bias_model)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.10.1a0.dev133+gb7284ab1-py3.6-macosx-10.7-x86_64.egg/jwst/lib/reffile_utils.py", line 344, in get_subarray_model
raise ValueError('Bad reference file slice indexes')
ValueError: Bad reference file slice indexes
```