search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
561 stars 167 forks source link

index out of bounds error for NRC_TSGRISM level_3 #7739

Closed stscijgbot-jp closed 1 year ago

stscijgbot-jp commented 1 year ago

Issue JP-3308 was created on JIRA by Hien Tran:

processing of new nrc_tsgrism level_3 dataset jw01550-c1001_20230717t083335_tso3_00002 (consisting of observations 1550:28 and 1550:29) with b9.2.1 gave the following error 


2023-07-17 09:17:39,554 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-07-17 09:17:39,565 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-07-17 09:17:39,689 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw01550028001_03103_00001-seg001_nrcalong_calints.fits>,).
2023-07-17 09:17:39,690 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': None, 'max_wavelength': None}
2023-07-17 09:17:39,698 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
2023-07-17 09:17:39,699 - stpipe.Tso3Pipeline - INFO - Extracting 1-D spectra ...
2023-07-17 09:17:39,812 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d running with args (<SlitModel(3, 64, 2048) from jw01550029001_03103_00001-seg001_nrcalong_calints.fits>,).
2023-07-17 09:17:39,814 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'extract_1d', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_atoca': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_wave_grid_in': None, 'soss_wave_grid_out': None, 'soss_estimate': None, 'soss_rtol': 0.0001, 'soss_max_grid_size': 20000, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'masking', 'soss_modelname': None}
2023-07-17 09:17:40,084 - CRDS - DEBUG - ========================================================================================================================
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() CRDS version: 11.16.22, b11.4.0, 64d96076d89b32a5687a6b77bb910ab93b3a99b3
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() observatory: jwst
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() reftypes: ('extract1d',)
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() context: None
2023-07-17 09:17:40,085 - CRDS - DEBUG - getreferences() ignore_cache: False
2023-07-17 09:17:40,085 - CRDS - DEBUG - CRDS_PATH = /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache
2023-07-17 09:17:40,085 - CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu
2023-07-17 09:17:40,086 - CRDS - DEBUG - Final effective context is 'jwst_1100.pmap'
2023-07-17 09:17:40,086 - CRDS - DEBUG - Computing best references locally.
2023-07-17 09:17:40,087 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.NOUTPUTS [NOUTPUTS]': '4.0',
'META.EXPOSURE.READPATT [READPATT]': 'RAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRC_TSGRISM',
'META.INSTRUMENT.CHANNEL [CHANNEL]': 'LONG',
'META.INSTRUMENT.CORONAGRAPH [CORONMSK]': 'UNDEFINED',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRCALONG',
'META.INSTRUMENT.FILTER [FILTER]': 'F356W',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'UNDEFINED',
'META.INSTRUMENT.MODULE [MODULE]': 'A',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRCAM',
'META.INSTRUMENT.PUPIL [PUPIL]': 'GRISMR',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-07-16',
'META.OBSERVATION.TIME [TIME-OBS]': '20:16:39.327',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBGRISM128',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2023-07-17 09:17:40,088 - stpipe.Tso3Pipeline.extract_1d - INFO - Using EXTRACT1D reference file N/A
2023-07-17 09:17:40,214 - CRDS - DEBUG - ========================================================================================================================
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() CRDS version: 11.16.22, b11.4.0, 64d96076d89b32a5687a6b77bb910ab93b3a99b3
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() observatory: jwst
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() reftypes: ('apcorr',)
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() context: None
2023-07-17 09:17:40,215 - CRDS - DEBUG - getreferences() ignore_cache: False
2023-07-17 09:17:40,215 - CRDS - DEBUG - CRDS_PATH = /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache
2023-07-17 09:17:40,215 - CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu
2023-07-17 09:17:40,215 - CRDS - DEBUG - Final effective context is 'jwst_1100.pmap'
2023-07-17 09:17:40,215 - CRDS - DEBUG - Computing best references locally.
2023-07-17 09:17:40,216 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.NOUTPUTS [NOUTPUTS]': '4.0',
'META.EXPOSURE.READPATT [READPATT]': 'RAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NRC_TSGRISM',
'META.INSTRUMENT.CHANNEL [CHANNEL]': 'LONG',
'META.INSTRUMENT.CORONAGRAPH [CORONMSK]': 'UNDEFINED',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NRCALONG',
'META.INSTRUMENT.FILTER [FILTER]': 'F356W',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'UNDEFINED',
'META.INSTRUMENT.MODULE [MODULE]': 'A',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRCAM',
'META.INSTRUMENT.PUPIL [PUPIL]': 'GRISMR',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-07-16',
'META.OBSERVATION.TIME [TIME-OBS]': '20:16:39.327',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBGRISM128',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2023-07-17 09:17:40,217 - stpipe.Tso3Pipeline.extract_1d - INFO - APCORR reference file name is "N/A"
2023-07-17 09:17:40,217 - stpipe.Tso3Pipeline.extract_1d - INFO - APCORR will NOT be applied
2023-07-17 09:17:40,241 - stpipe.Tso3Pipeline.extract_1d - INFO - Processing spectral order 1
2023-07-17 09:17:40,241 - stpipe.Tso3Pipeline.extract_1d - INFO - Beginning loop over 3 integrations ...
2023-07-17 09:17:40,241 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 1
2023-07-17 09:17:40,243 - stpipe.Tso3Pipeline.extract_1d - INFO - Using extraction limits: xstart=0, xstop=2047, ystart=0, ystop=63
2023-07-17 09:17:40,638 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 2
2023-07-17 09:17:41,032 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 3
2023-07-17 09:17:41,420 - stpipe.Tso3Pipeline.extract_1d - INFO - All 3 integrations done
2023-07-17 09:17:41,450 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 173, in process
x1d_result.int_times[row[0] - 1] = row
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/fitsrec.py", line 574, in __setitem__
self.field(self.names[idx])[key] = value.field(self.names[idx])
IndexError: index 2 is out of bounds for axis 0 with size 2
2023198091741 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw01550-c1001_20230717t083335_tso3_00002_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.cfg FAILED (exit=1) on jw01550-c1001_20230717t083335_tso3_00002_asn.json."
2023198091741 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw01550-c1001_20230717t083335_tso3_00002_asn msg="strun exit status=1"
2023198091742-E-ERROR-calibration_wrapper.csh FAILED. Archiving will NOT be attempted.
 1 / 1  ```
it crashed right after +*completing*+ the extract_1d step but just before performing white-light photometry for jw015500{*}29{*}001 . it had successfully completed all steps for jw015500{*}28{*}001 earlier.

could be related to JP-3146?
stscijgbot-jp commented 1 year ago

Comment by Maria Pena-Guerrero on JIRA:

I downloaded the files from MAST and ran TSO3 on observation 29 without issue. I am using pipeline version 1.12.0. Are you still seeing any issues?

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

Reproduced the failure with build 9.3.1 on August 30, 2023

stscijgbot-jp commented 1 year ago

Comment by Maria Pena-Guerrero on JIRA:

I just tested locally with build 9.3.1 and both CRDS contexts 1110 and 1130, and the TSO3 pipeline ran successfully on both. Was the data recently downloaded from MAST? I ran the TSO3 with the products directly from MAST, i.e. I did not do the whole reprocessing from uncal. 

stscijgbot-jp commented 1 year ago

Comment by John Scott on JIRA:

Sorry for the confusion - the problem is still showing up for jw01550-c1001_20230830t034805_tso3_00001 while this ticket references jw01550-c1001_20230717t083335_tso3_00002. On Aug 30, this 00002 product ran just fine, while the 00001 product showed the out-of-bounds error. 

stscijgbot-jp commented 1 year ago

Comment by Hien Tran on JIRA:

for reference, jw01550-c1001_20230717t083335_tso3_00002 is the same thing as jw01550-c1001_20230830t034805_tso3_00001. they both are associations for product named "jw01550-c1001_t005_nircam_f356w-grismr-subgrism128" consisting of two observations 28 & 29. so the failure is for the same product. 

the members in the association are:


                    "expname": "jw01550028001_03103_00001-seg001_nrcalong_calints.fits",
                    "exptype": "science",
                    "expname": "jw01550029001_03103_00001-seg001_nrcalong_calints.fits",
                    "exptype": "science",
                    "expname": "jw01550028001_02102_00001-seg001_nrcalong_cal.fits",
                    "exptype": "target_acquisition",
                    "expname": "jw01550029001_02102_00001-seg001_nrcalong_cal.fits",
                    "exptype": "target_acquisition", ```
the latest asn table file is attached. [Maria Pena-Guerrero](https://jira.stsci.edu/secure/ViewProfile.jspa?name=pena) did you run tso3 on this specific asn? 
stscijgbot-jp commented 1 year ago

Comment by Maria Pena-Guerrero on JIRA:

no, I did not run it on that specific asn. I will try the one you uploaded. I used jw01550-o029_20230830t034805_tso3_00001_asn.json, which has these:

                    "expname": "jw01550029001_03103_00001-seg001_nrcalong_calints.fits",

                    "exptype": "science",

                    "exposerr": "null",

                    "asn_candidate": "('o029', 'observation')"

                },

                {

                    "expname": "jw01550029001_02102_00001-seg001_nrcalong_cal.fits",

                    "exptype": "target_acquisition",

                    "exposerr": "null",

                    "asn_candidate": "[('o029', 'observation'), ('c1001', 'group')]"

stscijgbot-jp commented 1 year ago

Comment by Maria Pena-Guerrero on JIRA:

ok, I am now able to reproduce the error, yay! Investigating

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Investigating the content of this tso3 ASN. The problem in the Cal pipeline is being caused by the fact that data from two different observations are being included in the tso3: obs28 and obs29. Looking at the APT for program 1550, obs28 and obs29 are essentially identical in that they observe the same target, both in the NIRCam Time-Series Grism mode, using the SUBGRISM128 subarray. The only difference between the two is that they use 1 vs. 4 readout amps, leading to different readout times. Furthermore, there is a Special Requirement (SR) attached to both observations: "Group Observations 28, 29, non-interruptible". I suspect this is the reason why they were given the same asn-candidate ID, which then leads to them getting included in the same tso3 ASN by the asn generator.

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Regarding the asn_candidate ID's, here are the entries from the tso3 asn file:


                {
                    "expname": "jw01550028001_03103_00001-seg001_nrcalong_calints.fits",
                    "exptype": "science",
                    "exposerr": "null",
                    "asn_candidate": "[('o028', 'observation'), ('c1001', 'group')]"
                },
                {
                    "expname": "jw01550029001_03103_00001-seg001_nrcalong_calints.fits",
                    "exptype": "science",
                    "exposerr": "null",
                    "asn_candidate": "[('o029', 'observation'), ('c1001', 'group')]"
                },```
As you can see, each science member has a unique "observation" candidate ID (o028 vs. o029), but they are both assigned the same "group" candidate ID (c1001). Hence they both end up in the same c1001_tso3 ASN file, which in turn leads to the failure in calwebb_tso3 processing, because the algorithms are not anticipating data spread across multiple observations (nor should they be).

Furthermore, given that the two observations use slightly different readout parameters, there's no way anyone would **want** the data from the two combined together into a single level-3 product. They should receive independent processing.
stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

So the ultimate solution to this error will likely be in the form of an ASN rule update, so that multiple TSO observations don't get included in a single level-3 ASN.

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

I've download the jw01550 pool file from MAST and put it in the (newly created) data directory ██████████████████████████████████████████ so that it can be used for exploring ASN rule updates, if necessary.

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Jonathan Eisenhamer Given that the exposures from obs28 and obs29 are both assigned a c1001 group candidate ID, is it feasible to make some kind of update in the ASN rules to essentially just ignore the c1001 candidate (and hence not create a c1001 asn), and instead use only the o028 and o029 group candidate IDs to create individual observation-level asn's for each?

stscijgbot-jp commented 1 year ago

Comment by Tyler Pauly on JIRA:

I think it should be doable like this PR: https://github.com/spacetelescope/jwst/pull/6131/files

Is it true that no group candidates should ever be considered?

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Fixed by #7982 which modifies level-3 TSO ASN rules to no longer create any group-level ASN's for TSO's. So the c1001 tso3 asn that caused this problem won't get created anymore.

stscijgbot-jp commented 7 months ago

Comment by Hien Tran on JIRA:

ops has not had a chance reprocess this program with b10.0 yet. will verify next time it is reproed

stscijgbot-jp commented 2 months ago

Comment by Hien Tran on JIRA:

verified new behavior in ops