spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
568 stars 167 forks source link

fixed-type tuple error for level_3 tso data #7666

Closed stscijgbot-jp closed 1 year ago

stscijgbot-jp commented 1 year ago

Issue JP-3277 was created on JIRA by Michael Ridgaway:

During build reprocessing following the install of JWST DMS Build 9.2.1 the following error was seen during Level 3 processing of program 1224 and 2149 data. "ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int."

Logs can be found here:

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01224-o002_20230528t111952_tso3_00001_1685490152.276564

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02149-o002_20230528t221201_tso3_00001_1685336491.324613

 

This is the same error seen in JP-2900 that was fixed for nrc_wfss data.

stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

also seeing this for jw02158-o001_20230528t222142_tso3_00001

Directory Path: /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02158-o001_20230528t222142_tso3_00001_1685438395.977274

ALOG_1685438945_repro_level_3_jw02158-o001_20230528t222142_tso3_00001.out


2023-05-30 09:29:18,264 - CRDS - DEBUG - Computing best references locally.
2023-05-30 09:29:18,266 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.GRATING [GRATING]': 'PRISM',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-03-11',
'META.OBSERVATION.TIME [TIME-OBS]': '03:17:05.936',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB512',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '512.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1025.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '959.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....
<snip>

2023-05-31 03:25:36,867 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-05-31 03:32:35,517 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-05-31 03:32:54,735 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02158001001_04101_00001-seg003_nrs1_calints.fits>,).
2023-05-31 03:32:54,736 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-05-31 03:32:55,193 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
    step.run(*positional)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
    step_result = self.process(*args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
    self.save_model(x1d_result, suffix='x1dints')
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
    output_path = model.save(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
    self.to_fits(output_path, *args, **kwargs)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
    hdulist = fits_support.to_fits(self._instance, self._schema)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
    hdulist.append(_create_asdf_hdu(tree))
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
    column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
    array = self._convert_to_valid_data_type(array)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
    dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int.
2023155030646 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.cfg FAILED (exit=1) on jw02158-o001_20230528t222142_tso3_00001_asn.json."
2023155030646 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw02158-o001_20230528t222142_tso3_00001_asn msg="strun exit status=1" ```
stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

same problem seen for jw02508-o001_20230527t034736_tso3_00001, which is a MIR_LRS-SLITLESS tso3. so the problem seems to be connected to time series for all different inst/modes 

stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

another case, jw02488-o001_20230528t230213_tso3_00001, which is a NRS_BRIGHTOBJ tso

stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

many failure cases (exited = 1) for 2512, but some also succeeded (exit code 0). these are all nrs_brightobj tso. don't see any obvious differences between them, except the failure seems occur for those asns with more than 6 science input members


jw02512-c1002_20230603t201124_tso3_00001           REPRO_LEVEL_3        2023-06-04 01:31:14  2023-06-05 00:39:58  Exited          1          owlmgr             
jw02512-c1004_20230603t201124_tso3_00001           REPRO_LEVEL_3        2023-06-06 15:23:35  2023-06-08 06:44:35  Exited          1          owlmgr             
jw02512-o008_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:12:59  2023-06-04 22:17:47  Exited          1          owlmgr             
jw02512-o009_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:11:14  2023-06-04 17:01:07  Exited          1          owlmgr             
jw02512-o010_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:01:50  2023-06-04 15:23:49  Exited          1          owlmgr             
jw02512-o003_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:31:09  2023-06-04 11:29:52  Exited          0          owlmgr             
jw02512-o004_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:12:59  2023-06-04 10:25:02  Exited          0          owlmgr             
jw02512-o006_20230603t201124_tso3_00001            REPRO_LEVEL_3        2023-06-04 01:00:45  2023-06-04 09:06:53  Exited          0          owlmgr             
jw02512-o013_20230603t201124_tso3_00001            LEVEL_3              2023-06-05 23:11:15  2023-06-06 07:25:31  Exited          0          owlmgr       ```
stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Logs show that failure is occurring during the white_light step, after having successfully completed all steps through extract_1d.

stscijgbot-jp commented 1 year ago

Comment by Michael Ridgaway on JIRA:

We are also seeing this error occur for program 1201 data. Logs can be found here:

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01201-o002_20230612t154303_tso3_00001_1686632977.502441

stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

latest failure for jw02512-c1005_20230603t201124_tso3_00001, a 18-science member nrs_brightobj tso3 with sub2048 array


023-06-28 09:31:01,955 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NRSRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRS_BRIGHTOBJ',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRS1',
'META.INSTRUMENT.FILTER [FILTER]': 'F290LP',
'META.INSTRUMENT.GRATING [GRATING]': 'G395H',
'META.INSTRUMENT.LAMP_MODE [OPMODE]': 'BRIGHTOBJ',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRSPEC',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-06-27',
'META.OBSERVATION.TIME [TIME-OBS]': '04:55:57.198',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB2048',
'META.SUBARRAY.XSIZE [SUBSIZE1]': '2048.0',
'META.SUBARRAY.XSTART [SUBSTRT1]': '1.0',
'META.SUBARRAY.YSIZE [SUBSIZE2]': '32.0',
'META.SUBARRAY.YSTART [SUBSTRT2]': '965.0',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
....

2023-06-28 10:32:28,321 - stpipe.Tso3Pipeline.extract_1d - INFO - All 1525 integrations done
2023-06-28 10:32:38,739 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-06-28 10:33:16,877 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-06-28 10:33:24,340 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02512012001_04102_00001-seg002_nrs1_calints.fits>,).
2023-06-28 10:33:24,345 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-06-28 10:33:24,590 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 193, in process
self.save_model(x1d_result, suffix='x1dints')
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 978, in save_model
output_path = model.save(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 530, in save
self.to_fits(output_path, *args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/model_base.py", line 642, in to_fits
hdulist = fits_support.to_fits(self._instance, self._schema)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 494, in to_fits
hdulist.append(_create_asdf_hdu(tree))
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 506, in _create_asdf_hdu
column = fits.Column(array=data, format=fmt, name="ASDF_METADATA")
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 726, in __init__
array = self._convert_to_valid_data_type(array)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/column.py", line 1451, in _convert_to_valid_data_type
dtype = np.dtype(numpy_format).base
ValueError: invalid shape in fixed-type tuple: dimension does not fit into a C int. ```
stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

a list of affected datasets (cleaning these out of the pipeline for now)

jw01274-o004_20230531t014455_tso3_00002

jw01224-o002_20230528t111952_tso3_00001

jw02512-o009_20230603t201124_tso3_00001

jw02512-o008_20230603t201124_tso3_00001

jw02488-o001_20230528t230213_tso3_00001

jw02508-o001_20230527t034736_tso3_00001

jw02512-c1003_20230603t201124_tso3_00001

jw01633-o005_20230601t145410_tso3_00001

jw02149-o002_20230528t221201_tso3_00001

jw01952-o001_20230602t204759_tso3_00001

jw01633-o004_20230601t145410_tso3_00001

jw02512-c1004_20230603t201124_tso3_00001

jw02512-c1002_20230603t201124_tso3_00001

jw02512-c1006_20230603t201124_tso3_00001

jw02158-o001_20230528t222142_tso3_00001

jw01274-o003_20230531t014455_tso3_00002

jw01274-o002_20230531t014455_tso3_00002

jw01274-o001_20230531t014455_tso3_00002

jw01201-o002_20230612t154303_tso3_00001

jw02512-c1005_20230603t201124_tso3_00001

jw02512-o010_20230603t201124_tso3_00001

jw02512-o017_20230603t201124_tso3_00001

jw02159-o001_20230706t061547_tso3_00001

jw02512-c1003_20230603t201124_tso3_00001

stscijgbot-jp commented 1 year ago

Comment by Tyler Pauly on JIRA:

Quick investigation with a successful tso3 run, using a truncated input association, indicates that the ASDF extension takes up more memory than the other ~6098 extract1d extensions. Sending to ASDF experts - Brett Graham  maybe?

stscijgbot-jp commented 1 year ago

Comment by Brett Graham on JIRA:

Tyler Pauly do you have an easily sharable link to the hopefully similar and successful tso3 run? I can take a look at the file. My first concern would be that the extensions are being duplicated again for some reason (similar to https://github.com/asdf-format/asdf/issues/1232).

hbushouse commented 1 year ago

Fixed by https://github.com/spacetelescope/stdatamodels/pull/178

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Fixed by spacetelescope/stdatamodels#178

Will be included in final release candidate for B9.3 (jwst 1.11.2, B9.3rc3).

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

another case

 

jw01201-o002_20230612t154303_tso3_00001

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

jw02965-o003_20230709t121734_tso3_00001

is another case.

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

another

jw01185-o010_20230715t193546_tso3_00002 jw01185-o012_20230715t193546_tso3_00001

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

More (maybe some duplicates)

/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o005_20230724t181148_tso3_00001_1690242142.899445/ALOG_1690554631_repro_level_3_jw02084-o005_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o002_20230724t181148_tso3_00002_1690239819.034980/ALOG_1690559959_repro_level_3_jw02084-o002_20230724t181148_tso3_00002.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o001_20230724t181148_tso3_00001_1690239815.591388/ALOG_1690560200_repro_level_3_jw02084-o001_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o005_20230722t185115_tso3_00002_1690070138.815810/ALOG_1690070637_repro_level_3_jw01633-o005_20230722t185115_tso3_00002.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01633-o004_20230722t185115_tso3_00001_1690070141.316165/ALOG_1690070641_repro_level_3_jw01633-o004_20230722t185115_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-c1002_20230603t201124_tso3_00001_1690409345.284946/ALOG_1690409539_repro_level_3_jw02512-c1002_20230603t201124_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690554631_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02084-o003_20230724t181148_tso3_00001_1690244754.860707/ALOG_1690752134_repro_level_3_jw02084-o003_20230724t181148_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690554631_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01952-o001_20230724t114228_tso3_00001_1690213624.254566/ALOG_1690752143_repro_level_3_jw01952-o001_20230724t114228_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01274-o003_20230719t112441_tso3_00001_1689831012.394827/ALOG_1689960087_repro_level_3_jw01274-o003_20230719t112441_tso3_00001.err /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw02512-o018_20230603t201124_tso3_00001_1690223125.068266/ALOG_1690251734_level_3_jw02512-o018_202

stscijgbot-jp commented 8 months ago

Comment by Hien Tran on JIRA:

problem no longer seen with recent repro runs