search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

calwebb_spec3 error processing NRS_FIXEDSLIT #2070

Closed hbushouse closed 5 years ago

hbushouse commented 6 years ago

Mike Swam reports the following error in the calwebb_spec3 pipeline when testing SDP 7.1.3 and Cal 0.9.3rc1:

A single LEVEL_3 failure occurred for a spec_3 calibration attempt:
    /ifs/int/jwstc/store/rspencer/tests/SICDIL-Trimmed_OTIS-combined/cstring/JWSTDP-2018_1-180518/2018-05-19-020156/sdp/asn_creation/cal/level3/jw93045-o011_20180519t031641_spec3_001_asn.json
    (full log:)
strun /ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/jwst/pipeline//calwebb_spec3.cfg /ifs/int/jwstc/store/rspencer/tests/SICDIL-Trimmed_OTIS-combined/cstring/JWSTDP-2018_1-180518/2018-05-19-020156/sdp/asn_creation/cal/level3/jw93045-o011_20180519t031641_spec3_001_asn.json 
2018-05-19 14:50:05,419 - stpipe.Spec3Pipeline - INFO - Spec3Pipeline instance created. 
2018-05-19 14:50:05,422 - stpipe.Spec3Pipeline.mrs_imatch - INFO - MRSIMatchStep instance created. 
2018-05-19 14:50:05,424 - stpipe.Spec3Pipeline.cube_build - INFO - CubeBuildStep instance created. 
2018-05-19 14:50:05,425 - stpipe.Spec3Pipeline.extract_1d - INFO - Extract1dStep instance created. 
2018-05-19 14:50:05,427 - stpipe.Spec3Pipeline.resample_spec - INFO - ResampleSpecStep instance created. 
2018-05-19 14:50:05,428 - stpipe.Spec3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created. 
2018-05-19 14:50:05,429 - stpipe - INFO - Hostname: iljwdmscdp3 
2018-05-19 14:50:05,429 - stpipe - INFO - OS: Linux 
2018-05-19 14:50:05,521 - stpipe.Spec3Pipeline - INFO - Step Spec3Pipeline running with args ('/ifs/int/jwstc/store/rspencer/tests/SICDIL-Trimmed_OTIS-combined/cstring/JWSTDP-2018_1-180518/2018-05-19-020156/sdp/asn_creation/cal/level3/jw93045-o011_20180519t031641_spec3_001_asn.json',). 
2018-05-19 14:50:07,592 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02103_00001_nrs2_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Using CACHED CRDS reference assignment rules last updated on '2018-05-15 16:00:03.492657' 
CRDS - DEBUG - Using reference file selection rules 'jwst_0456.pmap' defined by cache. 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS2', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:25:22.340', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '980.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - READONLY CACHE skipping config update. 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:08,247 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:08,248 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:08,248 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:08,248 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:09,933 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02103_00001_nrs1_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:25:22.724', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '978.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:09,943 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:09,943 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:09,943 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:09,944 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:11,714 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02101_00001_nrs1_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:09:54.599', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '978.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:11,725 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:11,725 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:11,725 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:11,726 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:13,543 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02102_00001_nrs2_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS2', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:16:52.013', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '980.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:13,554 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:13,554 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:13,554 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:13,555 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:15,510 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02101_00001_nrs2_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS2', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:09:54.279', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '980.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:15,521 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:15,521 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:15,521 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:15,521 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:17,148 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw93045011001_02102_00001_nrs1_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol'] 
CRDS - DEBUG - ======================================================================================================================== 
CRDS - DEBUG - getreferences() CRDS version: 7.2.3, 7.2.3, f57ce29 
CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - getreferences() observatory: jwst 
CRDS - DEBUG - getreferences() reftypes: ('cubepar', 'drizpars', 'extract1d', 'resol') 
CRDS - DEBUG - getreferences() context: None 
CRDS - DEBUG - getreferences() ignore_cache: False 
CRDS - DEBUG - CRDS_PATH = /ifs/int/jwstc/ref/crds/cache 
CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu 
CRDS - DEBUG - Final effective context is 'jwst_0456.pmap' 
CRDS - DEBUG - Computing best references locally. 
CRDS - DEBUG - Bestrefs header: 
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID', 
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_FIXEDSLIT', 
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1', 
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP', 
'META.INSTRUMENT.GRATING [GRATING]': 'G395H', 
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', 
'META.OBSERVATION.DATE [DATE-OBS]': '2016-01-24', 
'META.OBSERVATION.TIME [TIME-OBS]': '05:16:52.333', 
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBS400A1', 
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', 
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', 
'META.SUBARRAY.YSIZE [SUBSIZE2]': '64.0', 
'META.SUBARRAY.YSTART [SUBSTRT2]': '978.0', 
'REFTYPE': 'UNDEFINED'} 
CRDS - DEBUG - Reference type 'extract1d' defined as 'jwst_nirspec_extract1d_0004.json' 
2018-05-19 14:50:17,159 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 
2018-05-19 14:50:17,159 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 
2018-05-19 14:50:17,159 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/ifs/int/jwstc/ref/crds/cache/references/jwst/nirspec/jwst_nirspec_extract1d_0004.json'. 
2018-05-19 14:50:17,160 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 
2018-05-19 14:50:17,160 - stpipe.Spec3Pipeline - INFO - Starting calwebb_spec3 ... 
2018-05-19 14:50:18,983 - stpipe.Spec3Pipeline - INFO - Convert from exposure-based to source-based data. 
Traceback (most recent call last): 
File "/ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/EGG-INFO/scripts/strun", line 14, in <module> 
step = Step.from_cmdline(sys.argv[1:]) 
File "/ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/jwst/stpipe/step.py", line 157, in from_cmdline 
return cmdline.step_from_cmdline(args) 
File "/ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline 
step.run(*positional) 
File "/ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/jwst/stpipe/step.py", line 391, in run 
step_result = self.process(*args) 
File "/ifs/int/jwstc/pipeline/pkgs/miniconda3/envs/jwstdp-0.9.3rc1-py35/lib/python3.5/site-packages/jwst-0.9.3rc1-py3.5-linux-x86_64.egg/jwst/pipeline/calwebb_spec3.py", line 106, in process 
output_file, source_id=int(source_id) 
ValueError: invalid literal for int() with base 10: 'S400A1'
hbushouse commented 6 years ago

We are not officially supporting calwebb_spec3 processing for slit-like spectra in B7.1.3, so this is not a high priority to get fixed in time for a 7.1.3 patch.

jdavies-st commented 5 years ago

Fixed by #2330.