search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
565 stars 167 forks source link

NIRSpec MSA failing calwebb_spec2 #3522

Closed stscijgbot closed 4 years ago

stscijgbot commented 5 years ago

Issue JP-733 was created by Hien Tran:

During NOPS5 20190522-20190524 several NIRSpec MSA exposures failed level_2b processing. The affected datasets are: |[jw00626-o030_20190523t122102_spec2_007|https://pljwdmsweb.stsci.edu/owlgui2/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o030_20190523t122102_spec2_007_1558696127.399148]| | |[jw00626-o030_20190523t122102_spec2_006|https://pljwdmsweb.stsci.edu/owlgui2/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o030_20190523t122102_spec2_006_1558696127.318253]| | |[jw00626-o030_20190523t122102_spec2_005|https://pljwdmsweb.stsci.edu/owlgui2/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o030_20190523t122102_spec2_005_1558695926.833931]| | |[jw00626-o030_20190523t122102_spec2_003|https://pljwdmsweb.stsci.edu/owlgui2/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o030_20190523t122102_spec2_003_1558695926.705074]|

Traceback excerpts from ALOG_1558696134_level_2b_jw00626-o030_20190523t122102_spec2_007.err:

{{CRDS - DEBUG - ======================================================================================================================== }} {{CRDS - DEBUG - getreferences() CRDS version: 7.2.9, 7.2.9, 5044611 }} {{CRDS - DEBUG - getreferences() server: [https://jwst-serverless-mode.stsci.edu|https://jwst-serverless-mode.stsci.edu/] }} {{CRDS - DEBUG - getreferences() observatory: jwst }} {{CRDS - DEBUG - getreferences() reftypes: ('ifuslicer',) }} {{CRDS - DEBUG - getreferences() context: None }} {{CRDS - DEBUG - getreferences() ignore_cache: False }} {{CRDS - DEBUG - CRDS_PATH = /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache }} {{CRDS - DEBUG - CRDS_SERVER_URL = [https://jwst-serverless-mode.stsci.edu|https://jwst-serverless-mode.stsci.edu/] }} {{CRDS - DEBUG - Final effective context is 'jwst_0525.pmap' }} {{CRDS - DEBUG - Computing best references locally. }} {{CRDS - DEBUG - Bestrefs header: }} {{{'META.EXPOSURE.READPATT [READPATT]': 'NRSIRS2RAPID', }} {{'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_MSASPEC', }} {{'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS2', }} {{'META.INSTRUMENT.FILTER [FILTER]': 'F170LP', }} {{'META.INSTRUMENT.GRATING [GRATING]': 'G235M', }} {{'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC', }} {{'META.OBSERVATION.DATE [DATE-OBS]': '2019-05-24', }} {{'META.OBSERVATION.TIME [TIME-OBS]': '09:25:02.358', }} {{'META.SUBARRAY.NAME [SUBARRAY]': 'FULL', }} {{'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0', }} {{'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0', }} {{'META.SUBARRAY.YSIZE [SUBSIZE2]': '2048.0', }} {{'META.SUBARRAY.YSTART [SUBSTRT2]': '1.0', }} {{'REFTYPE': 'UNDEFINED'} }} {{2019-05-24 11:08:59,217 - stpipe.Spec2Pipeline.assign_wcs - INFO - Retrieving open slitlets for msa_metadata_id = 5 }} {{2019-05-24 11:08:59,266 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<ImageModel(2048, 2048) from jw00626030001_02103_00001_nrs2_rate.fits>, ['/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02103_00002_nrs2_rate.fits', '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02103_00003_nrs2_rate.fits']). }} {{2019-05-24 11:09:02,785 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract done }} {{2019-05-24 11:09:02,785 - stpipe.Spec2Pipeline - {color:#FF0000}ERROR - Assign_wcs processing was skipped. {color}}} {color:#FF0000} {{Aborting remaining processing for this exposure. }}{color} {{No output product will be created. }} {{Traceback (most recent call last): }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 91, in process }} {{asn.filename }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 208, in process_exposure_product }} {{raise assign_wcs_exception }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 169, in process_exposure_product }} {{input = self.assign_wcs(input) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/stpipe/step.py", line 400, in run }} {{step_result = self.process(args) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/assign_wcs_step.py", line 80, in process }} {{result = load_wcs(input_model, reference_file_names) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/assign_wcs.py", line 53, in load_wcs }} {{pipeline = mod.create_pipeline(input_model, reference_files) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/nirspec.py", line 56, in create_pipeline }} {{pipeline = exp_type2transform[exp_type](input_model, reference_files) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/nirspec.py", line 250, in slits_wcs }} {{open_slits_id = get_open_slits(input_model, reference_files) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/nirspec.py", line 337, in get_open_slits }} {{slits = get_open_msa_slits(msa_metadata_file, msa_metadata_id) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/assign_wcs/nirspec.py", line 520, in get_open_msa_slits }} {{"sources for metadata_id = {}".format(msa_metadata_id)) }} {color:#FF0000}{{ValueError: MSA configuration file has more than 1 shutter with sources for metadata_id = 5 }}{color} {{Traceback (most recent call last): }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/EGG-INFO/scripts/strun", line 26, in  }} {{step = Step.from_cmdline(sys.argv[1:]) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/stpipe/step.py", line 162, in from_cmdline }} {{return cmdline.step_from_cmdline(args) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline }} {{step.run(positional) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/stpipe/step.py", line 400, in run }} {{step_result = self.process(*args) }} {{File "/jwdms/local/pipeline/pkgs/miniconda3/envs/jwstdp-0.12.3-py36/lib/python3.6/site-packages/jwst-0.12.3-py3.6-linux-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 107, in process }} {{'One or more products failed to process. Failing calibration.' }} {{RuntimeError: One or more products failed to process. Failing calibration. }} {{set savestatus = 1 }}

stscijgbot commented 5 years ago

Comment by Howard Bushouse: The above links to logs result in "File not found" errors. But looking at the one log that's displayed in the description, this is an expected error when using the B7.2.x pipeline, because the NIRSpec APT/MPT made changes ~6 months ago in the way sources are indicated in the MSA metadata file, which causes this error in Cal level-2b. This is fixed in the latest Cal and will be in B7.3. 

stscijgbot commented 5 years ago

Comment by Rosa Diaz: :D

stscijgbot commented 5 years ago

Comment by Rosa Diaz: When is this data going to be re-processed and archived?

stscijgbot commented 5 years ago

Comment by Hien Tran: The data are probably not going to get re-processed until the fix is delivered in build 7.3.

stscicrawford commented 5 years ago

Closing as it will be ready to test

stscijgbot commented 5 years ago

Comment by Hien Tran: This dataset got reprocessed on the A-string with DMS 7.3, and this error no longer occurred, but the 2b calibration still failed with a different, "missing MSA" error: {noformat} 2019-08-09 04:40:50,573 - CRDS - DEBUG - CRDS_PATH = /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache  2019-08-09 04:40:50,573 - CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu  2019-08-09 04:40:50,573 - CRDS - DEBUG - Final effective context is 'jwst_0541.pmap'  2019-08-09 04:40:50,573 - CRDS - DEBUG - Computing best references locally.  2019-08-09 04:40:50,574 - CRDS - DEBUG - Bestrefs header:  2019-08-09 04:40:50,574 - CRDS - DEBUG - {'META.EXPOSURE.READPATT [READPATT]': 'NRSIRS2RAPID',  2019-08-09 04:40:50,574 - CRDS - DEBUG - 'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_MSASPEC',  2019-08-09 04:40:50,574 - CRDS - DEBUG - 'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS2',  2019-08-09 04:40:50,574 - CRDS - DEBUG - 'META.INSTRUMENT.FILTER [FILTER]': 'F170LP',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.INSTRUMENT.GRATING [GRATING]': 'G235M',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.OBSERVATION.DATE [DATE-OBS]': '2019-05-24',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.OBSERVATION.TIME [TIME-OBS]': '09:25:02.358',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.SUBARRAY.NAME [SUBARRAY]': 'FULL',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.SUBARRAY.YSIZE [SUBSIZE2]': '2048.0',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'META.SUBARRAY.YSTART [SUBSTRT2]': '1.0',  2019-08-09 04:40:50,575 - CRDS - DEBUG - 'REFTYPE': 'UNDEFINED'}  2019-08-09 04:40:50,578 - stpipe.Spec2Pipeline.assign_wcs - ERROR - Missing MSA meta (MSAMETFL) file /ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02_msa.fits  2019-08-09 04:40:50,578 - stpipe.Spec2Pipeline - ERROR - Assign_wcs processing was skipped.  2019-08-09 04:40:50,578 - stpipe.Spec2Pipeline - ERROR - Aborting remaining processing for this exposure.  2019-08-09 04:40:50,578 - stpipe.Spec2Pipeline - ERROR - No output product will be created.  Traceback (most recent call last):  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/nirspec.py", line 517, in get_open_msa_slits  msa_file = fits.open(msa_file)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 151, in fitsopen  lazy_load_hdus, kwargs)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 390, in fromfile  lazy_load_hdus=lazy_load_hdus, kwargs)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 1039, in _readfrom  fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/utils/decorators.py", line 521, in wrapper  return function(*args, **kwargs)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/file.py", line 178, in init  self._open_filename(fileobj, mode, overwrite)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/file.py", line 555, in _open_filename  self._file = fileobj_open(self.name, IO_FITS_MODES[mode])  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/astropy/io/fits/util.py", line 397, in fileobj_open  return open(filename, mode, buffering=0)  FileNotFoundError: [Errno 2] No such file or directory: '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02_msa.fits' 

During handling of the above exception, another exception occurred: 

Traceback (most recent call last):  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_spec2.py", line 91, in process  asn.filename  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_spec2.py", line 189, in process_exposure_product  raise assign_wcs_exception  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_spec2.py", line 170, in process_exposure_product  input = self.assign_wcs(input)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py", line 394, in run  step_result = self.process(args)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/assign_wcs_step.py", line 83, in process  result = load_wcs(input_model, reference_file_names, slit_y_range)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/assign_wcs.py", line 44, in load_wcs  pipeline = mod.create_pipeline(input_model, reference_files, slit_y_range=nrs_slit_y_range)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/nirspec.py", line 58, in create_pipeline  pipeline = exp_type2transform[exp_type](input_model, reference_files, slit_y_range=slit_y_range)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/nirspec.py", line 271, in slits_wcs  open_slits_id = get_open_slits(input_model, reference_files, slit_y_range)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/nirspec.py", line 360, in get_open_slits  slits = get_open_msa_slits(msa_metadata_file, msa_metadata_id, dither_point, slit_y_range)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/assign_wcs/nirspec.py", line 521, in get_open_msa_slits  raise MSAFileError(message)  jwst.assign_wcs.util.MSAFileError: Missing MSA meta (MSAMETFL) file /ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02_msa.fits  Traceback (most recent call last):  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/EGG-INFO/scripts/strun", line 26, in   step = Step.from_cmdline(sys.argv[1:])  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py", line 162, in from_cmdline  return cmdline.step_from_cmdline(args)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline  step.run(positional)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py", line 394, in run  step_result = self.process(*args)  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_spec2.py", line 107, in process  'One or more products failed to process. Failing calibration.'  RuntimeError: One or more products failed to process. Failing calibration.  set savestatus = 1  source /dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab/bin/ssb_python_off  setenv PATH /home/owlmgr/.local/bin:/dms/local/jwst/pipeline/owl/bin:/dms/local/jwst/pipeline/tools:/usr/local/Java/jdk11.0.1-64/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/owlmgr/defs/bin:/dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab/bin:/dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab/bin/linux:/dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab/bin/java:/dms/local/jwst/pipeline/pkgs/DADSTools/envs/dads/bin  rehash  setenv PYTHONPATH /home/owlmgr/defs/bin:/dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab/bin:/dms/local/jwst/pipeline/jwstdp/JWSTDP-2019.1.0-208~877508ab:/dms/local/jwst/pipeline/pkgs/htcondor-8.4.5/lib/python  setenv XDG_CONFIG_HOME /home/owlmgr/rehearsal_config/defs/sdp_python  setenv XDG_CACHE_HOME /home/owlmgr/rehearsal_config/defs/sdp_python  echo SSB python deactivated; OWL python ready for interactive use.  if ( 1 != 0 ) then  echo eval date '+%Y%j%H%M%S'-E-ERROR-/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-cal-env/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_spec2.cfg failed on jw00626-o030_20190808t210959_spec2_005_asn.json.  eval date +%Y%j%H%M%S  date +%Y%j%H%M%S  exit 1  set savestatus = 1 {noformat}

stscijgbot commented 5 years ago

Comment by Howard Bushouse: This processing run didn't even get as far as the previous one, because it couldn't find the MSA metadata file that it needs for level-2b processing. The file it was trying to find is: 

/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/jw00626030001_02_msa.fits

Will need to investigate why this file was not found (was it ever created, was it in the right place at the right time, ...)

stscijgbot commented 5 years ago

Comment by Hien Tran: Similar level_2b (spec2) failures are also seen for the following dataset in NOPS6 at the assign_wcs step with the error "_MSA configuration file has more than 1 shutter with sources for metadataid = xx", where xx can be 64, 1, 44. |[jw00626-o032_20190909t010000_spec2_008|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o032_20190909t010000_spec2_008_1567990931.628907]| |[jw00626-o032_20190909t010000_spec2_010|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o032_20190909t010000_spec2_010_1567990931.917452]| |[jw00626-o032_20190909t010000_spec2_004|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o032_20190909t010000_spec2_004_1567990911.032979]| |[jw00626-o032_20190909t010000_spec2_003|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o032_20190909t010000_spec2_003_1567990890.504492]| |[jw00626-o031_20190908t230500_spec2_021|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o031_20190908t230500_spec2_021_1567984468.491821]| |[jw00626-o031_20190908t230500_spec2_022|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o031_20190908t230500_spec2_022_1567984468.928244]| |[jw00626-o031_20190908t230500_spec2_006|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o031_20190908t230500_spec2_006_1567984345.252913]| |[jw00626-o031_20190908t230500_spec2_002|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw00626-o031_20190908t230500_spec2_002_1567984324.352154]|

stscijgbot commented 4 years ago

Comment by Howard Bushouse: Where can I find these nOps6 datasets? Looking on the A-string, in the level2a and level2b directories, I don't find any datasets for jw00626*. We need the data files in order to test and debug. I'll also need the corresponding _msa.fits files.

stscijgbot commented 4 years ago

Comment by Hien Tran: Similar level_2b (spec2) failures are also seen for the following dataset in NOPS6 at the assign_wcs step with the error "_MSA configuration file has more than 1 shutter with sources for metadataid = xx", where xx can be 64, 1, 44. |[jw00626-o032_20190909t010000_spec2_008|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/252/owlmgr_jw00626-o032_20190909t010000_spec2_008_1567990931.628907]| |[jw00626-o032_20190909t010000_spec2_010|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/252/owlmgr_jw00626-o032_20190909t010000_spec2_010_1567990931.917452]| |[jw00626-o032_20190909t010000_spec2_004|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/252/owlmgr_jw00626-o032_20190909t010000_spec2_004_1567990911.032979]| |[jw00626-o032_20190909t010000_spec2_003|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/252/owlmgr_jw00626-o032_20190909t010000_spec2_003_1567990890.504492]| |[jw00626-o031_20190908t230500_spec2_021|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/251/owlmgr_jw00626-o031_20190908t230500_spec2_021_1567984468.491821]| |[jw00626-o031_20190908t230500_spec2_022|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/251/owlmgr_jw00626-o031_20190908t230500_spec2_022_1567984468.928244]| |[jw00626-o031_20190908t230500_spec2_006|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/251/owlmgr_jw00626-o031_20190908t230500_spec2_006_1567984345.252913]| |[jw00626-o031_20190908t230500_spec2_002|https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs_old/2019/251/owlmgr_jw00626-o031_20190908t230500_spec2_002_1567984324.352154]|

stscijgbot commented 4 years ago

Comment by Howard Bushouse: The links in the comments above take me to log files, but we need the DATA files. When I go to the data path listed in the log files:

/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level2/

I find lots of level-2b data products and ASN files, but none of them is from program 626:

  {code:java}  ls jw00626*  ls: No match. {code}  

stscijgbot commented 4 years ago

Comment by Hien Tran: [~bushouse], the data got cleaned out after every rehearsal to prepare for and run the next one. Nops6 products are no longer there since we had to run GSEG3 EMTB DR1 after that. I think the best way to get the data is to go grab them from the portal [https://pwjwdmsauiweb.stsci.edu/portal/Mashup/Clients/Mast/Portal.html]

stscijgbot commented 4 years ago

Comment by Howard Bushouse: I can't find them in the portal either, likely due to the fact they failed in processing and hence were never ingested. I can find other exposures from the same observation, but those succeed in processing.

stscijgbot commented 4 years ago

Comment by Hien Tran: The level_2a data should be there.

Anyway, I found that C-string sees the same problem and we still have the data for it sitting on disk. I've packaged the 626-o032 fits and json files from the C-string /asn_creation/cal/level2/ area and put them in a tar ball at /grp/jwst/dms2/htran/msa626032.tar.gz. The spec2_003, _006, _009, _012 sets are the ones in trouble. All the others went through ok. 

stscijgbot commented 4 years ago

Comment by Howard Bushouse: Thanks [~htran]. I've tried processing the data files that you provided and have replicated the error. It appears to be an error in the front-end systems that generate the MSA meta data, such that there's one slitlet that has more than 1 source labeled as being primary. This causes the level-2b cal pipeline to crash. The NIRSpec APT/PPS folks have been notified about the error. But until the msa meta file contents get fixed, the pipeline will continue to crash.