Closed stscijgbot-jp closed 1 year ago
also seeing this for jw02158-o001_20230528t222142_tso3_00001
Directory Path: /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02158-o001_20230528t222142_tso3_00001_1685438395.977274
ALOG_1685438945_repro_level_3_jw02158-o001_20230528t222142_tso3_00001.out
2023-05-30 09:29:18,264 - CRDS - DEBUG - Computing best references locally.
2023-05-30 09:29:18,266 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.GRATING [GRATING]': 'PRISM',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-03-11',
'META.OBSERVATION.TIME [TIME-OBS]': '03:17:05.936',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB512',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '512.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1025.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '959.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....
<snip>
2023-05-31 03:25:36,867 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-05-31 03:32:35,517 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-05-31 03:32:54,735 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02158001001_04101_00001-seg003_nrs1_calints.fits>,).
2023-05-31 03:32:54,736 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-05-31 03:32:55,193 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
self.save_model(x1d_result, suffix='x1dints')
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
output_path = model.save(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
self.to_fits(output_path, *args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
hdulist = fits_support.to_fits(self._instance, self._schema)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
hdulist.append(_create_asdf_hdu(tree))
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
array = self._convert_to_valid_data_type(array)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int.
2023155030646 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.cfg FAILED (exit=1) on jw02158-o001_20230528t222142_tso3_00001_asn.json."
2023155030646 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun exit status=1" ```
same problem seen for jw02508-o001_20230527t034736_tso3_00001, which is a MIR_LRS-SLITLESS tso3. so the problem seems to be connected to time series for all different inst/modes
another case, jw02488-o001_20230528t230213_tso3_00001, which is a NRS_BRIGHTOBJ tso
many failure cases (exited = 1) for 2512, but some also succeeded (exit code 0). these are all nrs_brightobj tso. don't see any obvious differences between them, except the failure seems occur for those asns with more than 6 science input members
jw02512-c1002_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:31:14 2023-06-05 00:39:58 Exited 1 owlmgr
jw02512-c1004_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-06 15:23:35 2023-06-08 06:44:35 Exited 1 owlmgr
jw02512-o008_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:12:59 2023-06-04 22:17:47 Exited 1 owlmgr
jw02512-o009_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:11:14 2023-06-04 17:01:07 Exited 1 owlmgr
jw02512-o010_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:01:50 2023-06-04 15:23:49 Exited 1 owlmgr
jw02512-o003_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:31:09 2023-06-04 11:29:52 Exited 0 owlmgr
jw02512-o004_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:12:59 2023-06-04 10:25:02 Exited 0 owlmgr
jw02512-o006_20230603t201124_tso3_00001 REPRO_LEVEL_3 2023-06-04 01:00:45 2023-06-04 09:06:53 Exited 0 owlmgr
jw02512-o013_20230603t201124_tso3_00001 LEVEL_3 2023-06-05 23:11:15 2023-06-06 07:25:31 Exited 0 owlmgr ```
Comment by Howard Bushouse on JIRA:
Logs show that failure is occurring during the white_light step, after having successfully completed all steps through extract_1d.
Comment by Michael Ridgaway on JIRA:
We are also seeing this error occur for program 1201 data. Logs can be found here:
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01201-o002_20230612t154303_tso3_00001_1686632977.502441
latest failure for jw02512-c1005_20230603t201124_tso3_00001, a 18-science member nrs_brightobj tso3 with sub2048 array
023-06-28 09:31:01,955 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP',
'META.INSTRUMENT.GRATING [GRATING]': 'G395H',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-06-27',
'META.OBSERVATION.TIME [TIME-OBS]': '04:55:57.198',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB2048',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '965.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....
2023-06-28 10:32:28,321 - stpipe.Tso3Pipeline.extract_1d - INFO - All 1525 integrations done
2023-06-28 10:32:38,739 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-06-28 10:33:16,877 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-06-28 10:33:24,340 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02512012001_04102_00001-seg002_nrs1_calints.fits>,).
2023-06-28 10:33:24,345 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-06-28 10:33:24,590 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
self.save_model(x1d_result, suffix='x1dints')
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
output_path = model.save(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
self.to_fits(output_path, *args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
hdulist = fits_support.to_fits(self._instance, self._schema)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
hdulist.append(_create_asdf_hdu(tree))
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
array = self._convert_to_valid_data_type(array)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int. ```
Comment by John Scott on JIRA:
a list of affected datasets (cleaning these out of the pipeline for now)
jw01274-o004_20230531t014455_tso3_00002
jw01224-o002_20230528t111952_tso3_00001
jw02512-o009_20230603t201124_tso3_00001
jw02512-o008_20230603t201124_tso3_00001
jw02488-o001_20230528t230213_tso3_00001
jw02508-o001_20230527t034736_tso3_00001
jw02512-c1003_20230603t201124_tso3_00001
jw01633-o005_20230601t145410_tso3_00001
jw02149-o002_20230528t221201_tso3_00001
jw01952-o001_20230602t204759_tso3_00001
jw01633-o004_20230601t145410_tso3_00001
jw02512-c1004_20230603t201124_tso3_00001
jw02512-c1002_20230603t201124_tso3_00001
jw02512-c1006_20230603t201124_tso3_00001
jw02158-o001_20230528t222142_tso3_00001
jw01274-o003_20230531t014455_tso3_00002
jw01274-o002_20230531t014455_tso3_00002
jw01274-o001_20230531t014455_tso3_00002
jw01201-o002_20230612t154303_tso3_00001
jw02512-c1005_20230603t201124_tso3_00001
jw02512-o010_20230603t201124_tso3_00001
jw02512-o017_20230603t201124_tso3_00001
jw02159-o001_20230706t061547_tso3_00001
jw02512-c1003_20230603t201124_tso3_00001
Comment by Tyler Pauly on JIRA:
Quick investigation with a successful tso3 run, using a truncated input association, indicates that the ASDF extension takes up more memory than the other ~6098 extract1d extensions. Sending to ASDF experts - Brett Graham maybe?
Comment by Brett Graham on JIRA:
Tyler Pauly do you have an easily sharable link to the hopefully similar and successful tso3 run? I can take a look at the file. My first concern would be that the extensions are being duplicated again for some reason (similar to https://github.com/asdf-format/asdf/issues/1232).
Comment by Howard Bushouse on JIRA:
Fixed by spacetelescope/stdatamodels#178
Will be included in final release candidate for B9.3 (jwst 1.11.2, B9.3rc3).
Comment by John Scott on JIRA:
another
jw01185-o010_20230715t193546_tso3_00002 jw01185-o012_20230715t193546_tso3_00001
Comment by John Scott on JIRA:
More (maybe some duplicates)
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o005_20230724t181148_tso3_00001_1690242142.899445/ALOG_1690554631_repro_level_3_jw02084-o005_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o002_20230724t181148_tso3_00002_1690239819.034980/ALOG_1690559959_repro_level_3_jw02084-o002_20230724t181148_tso3_00002.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o001_20230724t181148_tso3_00001_1690239815.591388/ALOG_1690560200_repro_level_3_jw02084-o001_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o005_20230722t185115_tso3_00002_1690070138.815810/ALOG_1690070637_repro_level_3_jw01633-o005_20230722t185115_tso3_00002.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o004_20230722t185115_tso3_00001_1690070141.316165/ALOG_1690070641_repro_level_3_jw01633-o004_20230722t185115_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-c1002_20230603t201124_tso3_00001_1690409345.284946/ALOG_1690409539_repro_level_3_jw02512-c1002_20230603t201124_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690554631_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690752134_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690554631_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690752143_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01274-o003_20230719t112441_tso3_00001_1689831012.394827/ALOG_1689960087_repro_level_3_jw01274-o003_20230719t112441_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-o018_20230603t201124_tso3_00001_1690223125.068266/ALOG_1690251734_level_3_jw02512-o018_202
Issue JP-3277 was created on JIRA by Michael Ridgaway:
During build reprocessing following the install of JWST DMS Build 9.2.1 the following error was seen during Level 3 processing of program 1224 and 2149 data. "ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int."
Logs can be found here:
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01224-o002_20230528t111952_tso3_00001_1685490152.276564
/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02149-o002_20230528t221201_tso3_00001_1685336491.324613
This is the same error seen in JP-2900 that was fixed for nrc_wfss data.