index out of bounds error for MIR_LRS-SLITLESS

[stscijgbot-jp]

Issue JP-3146 was created on JIRA by Hien Tran:

repro of program 2783 from l1a with b9.1.2 is giving the following tso3 error when processing a MIR_LRS-SLITLESS dataset jw02783-c1000_20230320t092349_tso3_00001

2023-03-22 00:23:41,313 - stpipe.Tso3Pipeline.extract_1d - INFO - All 10 integrations done
2023-03-22 00:23:41,372 - CRDS - DEBUG - Using reference file selection rules 'jwst_1069.pmap' defined by cache.
2023-03-22 00:23:41,373 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
2023-03-22 00:23:41,414 - stpipe.Tso3Pipeline - INFO - Performing white-light photometry ...
2023-03-22 00:23:41,780 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light running with args (<MultiSpecModel from jw02783002001_03103_00001-seg001_mirimage_calints.fits>,).
2023-03-22 00:23:41,781 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.ecsv', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'whtlt', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'min_wavelength': 5.0, 'max_wavelength': 12.0}
2023-03-22 00:23:41,790 - stpipe.Tso3Pipeline.white_light - INFO - Step white_light done
2023-03-22 00:23:41,790 - stpipe.Tso3Pipeline - INFO - Extracting 1-D spectra ...
2023-03-22 00:23:42,142 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d running with args (<CubeModel(200, 416, 72) from jw02783001001_04103_00001-seg002_mirimage_calints.fits>,).
2023-03-22 00:23:42,144 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'extract_1d', 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_atoca': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_wave_grid_in': None, 'soss_wave_grid_out': None, 'soss_estimate': None, 'soss_rtol': 0.0001, 'soss_max_grid_size': 20000, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'masking', 'soss_modelname': None}
2023-03-22 00:23:42,163 - CRDS - DEBUG - ========================================================================================================================
2023-03-22 00:23:42,163 - CRDS - DEBUG - getreferences() CRDS version: 11.16.19, b11.4.0, 64d96076d89b32a5687a6b77bb910ab93b3a99b3
2023-03-22 00:23:42,163 - CRDS - DEBUG - getreferences() server: https://jwst-serverless-mode.stsci.edu
2023-03-22 00:23:42,163 - CRDS - DEBUG - getreferences() observatory: jwst
2023-03-22 00:23:42,163 - CRDS - DEBUG - getreferences() reftypes: ('extract1d',)
2023-03-22 00:23:42,163 - CRDS - DEBUG - getreferences() context: None
2023-03-22 00:23:42,164 - CRDS - DEBUG - getreferences() ignore_cache: False
2023-03-22 00:23:42,164 - CRDS - DEBUG - CRDS_PATH = /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache
2023-03-22 00:23:42,164 - CRDS - DEBUG - CRDS_SERVER_URL = https://jwst-serverless-mode.stsci.edu

2023-03-22 00:23:42,177 - CRDS - DEBUG - Final effective context is 'jwst_1069.pmap'
2023-03-22 00:23:42,177 - CRDS - DEBUG - Computing best references locally.
2023-03-22 00:23:42,178 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'FASTR1',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'MIR_LRS-SLITLESS',
'META.INSTRUMENT.BAND [BAND]': 'UNDEFINED',
'META.INSTRUMENT.CHANNEL [CHANNEL]': 'UNDEFINED',
'META.INSTRUMENT.CORONAGRAPH [CORONMSK]': 'UNDEFINED',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'MIRIMAGE',
'META.INSTRUMENT.FILTER [FILTER]': 'P750L',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'OFF',
'META.INSTRUMENT.NAME [INSTRUME]': 'MIRI',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-02-14',
'META.OBSERVATION.TIME [TIME-OBS]': '15:03:20.312',
'META.SUBARRAY.NAME [SUBARRAY]': 'SLITLESSPRISM',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2023-03-22 00:23:42,178 - CRDS - DEBUG - Reference type 'apcorr' defined as 'jwst_miri_apcorr_0007.fits'
2023-03-22 00:23:42,179 - stpipe.Tso3Pipeline.extract_1d - INFO - Using APCORR file /ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/miri/jwst_miri_apcorr_0007.fits
2023-03-22 00:23:42,226 - stpipe.Tso3Pipeline.extract_1d - WARNING - spectral_order is None; using 1
2023-03-22 00:23:42,226 - stpipe.Tso3Pipeline.extract_1d - INFO - Processing spectral order 1
2023-03-22 00:23:42,226 - stpipe.Tso3Pipeline.extract_1d - INFO - Beginning loop over 200 integrations ...
2023-03-22 00:23:42,226 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 1
2023-03-22 00:23:42,228 - stpipe.Tso3Pipeline.extract_1d - INFO - Using extraction limits: xstart=32.5, xstop=42.5, ystart=6, ystop=395
2023-03-22 00:23:42,228 - stpipe.Tso3Pipeline.extract_1d - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/jwst/extract_1d/extract.py:1888: RuntimeWarning: Mean of empty slice
wavelength = np.nanmean(wl[sy0:sy1, sx0:sx1], axis=1)

2023-03-22 00:23:42,330 - stpipe.Tso3Pipeline.extract_1d - INFO - Applying Aperture correction.
2023-03-22 00:23:42,575 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 2
2023-03-22 00:23:42,679 - stpipe.Tso3Pipeline.extract_1d - INFO - Applying Aperture correction.
2023-03-22 00:23:42,928 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 3
2023-03-22 00:23:43,031 - stpipe.Tso3Pipeline.extract_1d - INFO - Applying Aperture correction.
...
<snip>

2023-03-22 00:24:52,835 - stpipe.Tso3Pipeline.extract_1d - INFO - Extracting integration 200
2023-03-22 00:24:52,937 - stpipe.Tso3Pipeline.extract_1d - INFO - Applying Aperture correction.
2023-03-22 00:24:53,182 - stpipe.Tso3Pipeline.extract_1d - INFO - All 200 integrations done
2023-03-22 00:24:54,239 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/bin/strun", line 28, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 170, in process
x1d_result.int_times[row[0] - 1] = row
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/astropy/io/fits/fitsrec.py", line 574, in __setitem__
self.field(self.names[idx])[key] = value.field(self.names[idx])
IndexError: index 200 is out of bounds for axis 0 with size 10
----------------------------------------------------------------------
ERROR RUNNING STEP 'Tso3Pipeline':
index 200 is out of bounds for axis 0 with size 10
----------------------------------------------------------------------
2023081002455-E-ERROR-/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.9.6.20230309-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.cfg FAILED (exit=1) on jw02783-c1000_20230320t092349_tso3_00001_asn.json.
2023081002455-E-ERROR-exit status=1 ```
it crashed right after +*completing*+ the extract_1d step but just before performing white-light photometry. it had successfully completed all steps for another segment earlier.

we had l3 products successfully made for the same dataset with the previous build. 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Tyler Pauly I'm downloading the data for this right now from MAST and will put in ██████████████████████████████████████████ when I've got them. Given that the level 2 and 3 products in MAST right now are from 1.8.2 processing, I'm retrieving the uncal files, so that we can reprocess with 1.9.6 all the way from the start, just in case anything else upstream leads to changes that would not make this a fair test.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Hien Tran Looking more carefully at the log above, I'm confused. First I see "jw02783002001_03103_00001-seg001_mirimage_calints.fits" being processed successfully through the extract_1d and white_light steps, but then it fails when processing "jw02783_001001_4103_00001-seg002_mirimage_calints.fits". Those two datasets are from different observations (obs-002 and obs-001) and also have different activity id's. On the surface it seems to me like those don't belong together in the same level 3 association. So maybe that's the problem (in the generation of the tso3 ASN?). We'll check.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

The crash is a result of code added for JP-2677, which made an assumption that all science models in the input association are segments corresponding to one int_times table. The input c1000_tso3 association here has all segments from obs 001 but also a segment from o002. The pool states that both observations are science, and also pairs them in a group association candidate (c1000), while APT shows obs 002 is listed as a background observation. I would suggest that we ask APT/MIRI team/program PI if obs002 should be labelled as a background observation, such that its spectra are not included in the final output x1dints file.

It's not clear if tso3 is working improperly - is it expected that we'll have group-level association candidates where we are merging observations with separate int_times tables into a final product with entries from both/multiple int_times? This may be the question for MIRI team/CalWG.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Observation 2 is a dedicated background, which we are now recommending that people take with LRS slitless observations. The pipeline shouldn't try to subtract this from the science data though, and it should not be folded into the science data product (not sure if I'm interpreting Tyler Pauly 's comment above correctly). Apologies, we should have checked with the pipeline whether this would cause issues. 

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Hi Sarah Kendrew , my understanding of the remaining issue (which may be limited) is that observation 2, labeled "background observation" in APT, is not pointing to a target that is defined as a background. Because the target is not defined as a BKGDTARG, no background labels will be applied to exposures in associations for processing. The pool file will not show any observations as backgrounds, because the label field of the observation in APT is not carried forward. 

The pipeline issue will likely continue unless a second target is defined which is labeled as a background target in the program. I think further questions should maybe be directed towards APT, as the pipeline problem is really a downstream effect of the APT program settings.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

OK thanks Tyler Pauly . How big of an issue is it if we do nothing? I think it's fine for these observations to not go through level 3 processing, but if every time the error arises it creates pesky work for someone, then we can obv find a workaround. 

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

looks like we have another case of this issue with new data jw02965-c1000_20230709t121734_tso3_00001, a MIR_LRS-SLITLESS tso3 asn consisting of obs 1 (target) & 2  (bkgrn), with 2 defined as a GROUP association candidate, not BACKGROUND.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Tyler Pauly do you have a copy of the pool file for this program? If so, could you put it in █████████████████████████████████████████ along with the other data files for this?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The only obvious difference in meta data between the obs-1 and obs-2 data are the XOFFSET/YOFFSET values, due to the use of the special requirement in obs-2 to move the target out of the aperture. Unfortunately those particular values are not included in the pool files and hence it may not be possible to use them (at least not easily) in updated ASN rules to exclude the obs-2 background exposure(s) from the level-3 ASN. Similarly, it would also not be easy to have APT exclude them from the c1000 candidate ID, because it too has no obvious indicators to suggest anything unusual about obs-2 (other than the SR with the offset).

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

i've copied the pool jw02965_20230709t121734_pool.csv into ███████████████████████████████████████

[stscijgbot-jp]

Comment by John Scott on JIRA:

We have another case, jw03548-c1000_20230724t111939_tso3_00001, which is a new observation processed with build 9.2.2

2023-07-24 14:33:52,497 - stpipe.Tso3Pipeline.extract_1d - INFO - Step extract_1d done 
Traceback (most recent call last): 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/bin/strun", line 28, in <module> 
step = Step.from_cmdline(sys.argv[1:]) 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline 
return cmdline.step_from_cmdline(args) 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline 
step.run(*positional) 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 454, in run 
step_result = self.process(*args) 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 173, in process 
x1d_result.int_times[row[0] - 1] = row 
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.10.1.20230413-py3.9/lib/python3.9/site-packages/astropy/io/fits/fitsrec.py", line 574, in __setitem__ 
self.field(self.names[idx])[key] = value.field(self.names[idx]) 
IndexError: index 165 is out of bounds for axis 0 with size 10  ```

[stscijgbot-jp]

Comment by John Scott on JIRA:

Also jw01274-c1000_20231114t122509_tso3_00001

Traceback (most recent call last):File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/bin/strun", line 26, in <module>step = Step.from_cmdline(sys.argv[1:])File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 186, in from_cmdlinereturn cmdline.step_from_cmdline(args)File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 386, in step_from_cmdlinestep.run(*positional)File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 478, in runstep_result = self.process(*args)File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso3.py", line 173, in processx1d_result.int_times[row[0] - 1] = rowFile "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/astropy/io/fits/fitsrec.py", line 567, in __setitem__self.field(self.names[idx])[key] = value.field(self.names[idx]) ```

[stscijgbot-jp]

Comment by David Law on JIRA:

I don't see a clear deficiency in the JDox documentation of how APT programs should be designed, but nonetheless these seem to have slipped through with APT incorrectly configured.  It's clear from a visual analysis of the APT program what the users intended, but it's not obvious that there's any metadata that could unambiguously communicate that.

Howard Bushouse Tyler Pauly Is there any way to 'hack' the metadata of a program after the fact to fix errors such as these?  I'm thinking of the SDSS solution to occasionally-corrupted header info, which was to maintain a list of exposures that had to be header-fixed prior to processing.  We could then hand-fix cases such as these when they come up by forcing (say) PID 1274 Obs 5 to be the background for Obs 4.

Thoughts?

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

For an observation to be the background, it needs to have the correct association candidate and it needs to have the BKGDTARG flag turned to True. For the candidate, I think that's probably in the APT/PPS realm. The same is true of BKGDTARG, but that part seems more 'hackable'.

If users want background observations, a target needs to be labeled as a background, i.e. BKGDTARG=True. If APT needs separate targets for this, i.e. the same target can't be considered science in one observation and a background in another, perhaps this is a feature request for APT?

[stscijgbot-jp]

Comment by David Law on JIRA:

PIDs 1274, 3548, 2783, and 2965 above all look like user error.  There's an easy way of doing what they wanted in APT, but they didn't follow guidelines in setting up the program and program technical review missed the error.

So, to what extent do we save users from their own mistakes?  Scientifically, I'd say it would be better if we hack BKGDTARG=True when we see this crash (after pinging the relevant instrument team to confirm whether that's a reasonable assumption for each new case).  Effort-wise, which is worse for DMS to deal with, having PIDs around that will fail out of the pipeline as above, or having to hack the BKGDTARG flags?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

David Law We'd have to check with SDP/DMS folks to find out if there's a way to permanently "hack" metadata values for a dataset, e.g. to change the setting of BKGDTARG. I think there's the possibility of the use of some kind of "override" files that get invoked whenever a given dataset is reprocessed, the contents of which tell it what changes to make to metadata. But yeah, how much effort is worth it to save users from themselves.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Hi Hien Tran and/or Jesse Doggett, do you have any thoughts or suggestions on the comments by Howard and David? Effectively, some programs will cause errors for operations/repro unless they are either excluded or "hacked" to have background flags corrected for some observations. Is there a preference on either option, or option 3: let the errors continue? Any suggestions on other options not yet voiced would also be welcome.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

If I understand correctly, hacking the BKGDTARG flags means modifying the PPS database. Essentially, I think, that is correcting the APT data in the PPS database.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Just to add that we are gradually updating all TSO programs to add the background target appropriately, so this won't happen. Making TA for slitless LRS has also simplified the implementation in APT, which will help. But I've had at least one that we missed due to already being set to flight-ready - sorry about that. This can also be resolved by fleshing out the background subtraction step in the pipeline for slitless LRS, which I believe is on team LRS' to do list. 

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

i think we can probably perform a hack on the dms side to change BKGDTARG=T for the background visit. we can give that a try and reprocess the affected program to see if it works.  

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

I'm really sorry, we have another observation coming up that will have this problem - PID 3557 Obs 9 + 16 (16 is the background). I asked the PI to implement the background differently, with a dedicated background target, but unfortunately as the target has been observed by a previous observation in the same program, it is now locked in APT. 

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

the test to hack the bkgdtarg keyword on the dms side (DMSOPS-1211) seems to work ok.  i can go ahead and apply the hack to the rest of the affected visits, however, thanks to JP-3308, we will never see this issue anymore. i don't know if anything else can be done on this ticket, so i think it can probably be closed. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Thanks Hien Tran ; closing this ticket out then.