1516:5 L2B failure in Spec2Pipeline.extract_1d

[stscijgbot-jp]

Issue JP-2929 was created on JIRA by Aiden Kovacs:

On 2022-07-10 19:53 UTC, jw01516-o005_20220710t191635_spec2_001 failed in L2B with the following error message:

(from  /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01516-o005_20220710t191635_spec2_001_1657482728.626343ALOG_1657496072_level_2b_jw01516-o005_20220710t191635_spec2_001.err)

2022-07-10 23:34:39,671 - CRDS - DEBUG - Reference type 'speckernel' defined as 'jwst_niriss_speckernel_0004.fits'
2022-07-10 23:34:39,890 - stpipe.Spec2Pipeline.extract_1d - INFO - Input is an ImageModel, processing a single integration.
2022-07-10 23:34:39,914 - stpipe.Spec2Pipeline.extract_1d - INFO - Solving for the transformation parameters.
2022-07-10 23:34:40,308 - stpipe.Spec2Pipeline.extract_1d - INFO - Measuring trace position for orders 1 and 2.
2022-07-10 23:34:40,706 - stpipe.Spec2Pipeline.extract_1d - INFO - Measured to Reference trace position transform: theta=0.1000, dx=-3.0161, dy=-4.6938
2022-07-10 23:34:46,739 - stpipe.Spec2Pipeline.extract_1d - INFO - Extracting using transformation parameters [ 0.09998556 -3.0160868 -4.69383471]
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 117, in process
result = self.process_exposure_product(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 298, in process_exposure_product
x1d = self.extract_1d(resampled)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/stpipe/step.py", line 430, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/extract_1d_step.py", line 356, in process
result, ref_outputs = soss_extract.run_extract1d(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 590, in run_extract1d
result = model_image(scidata_bkg, scierr, scimask, refmask, ref_file_args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 291, in model_image
engine = ExtractionEngine(*ref_file_args, n_os=n_os, threshold=threshold, c_kwargs=c_kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/atoca.py", line 1568, in __init__
super().__init__(wave_map, trace_profile, *args, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/atoca.py", line 177, in __init__
self.weights, self.weights_k_idx = self.compute_weights()
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/atoca.py", line 541, in compute_weights
weights_n, k_idx_n = self.get_w(i_order) # Compute weights
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.5.3.20220620-py39/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/atoca.py", line 1779, in get_w
nume2 = wave_grid[k_n[cond, n_ki - 1]] - wave_p[cond]
IndexError: index 5258 is out of bounds for axis 0 with size 5258 ```
It crashed right before “Solving for the optimal Tikhonov factor” - comparatively, here's a successful case from o004:
```java
2022-07-10 19:50:24,158 - stpipe.Spec2Pipeline.extract_1d - INFO - Solving for the transformation parameters.
2022-07-10 19:50:24,541 - stpipe.Spec2Pipeline.extract_1d - INFO - Measuring trace position for orders 1 and 2.
2022-07-10 19:50:24,947 - stpipe.Spec2Pipeline.extract_1d - INFO - Measured to Reference trace position transform: theta=0.2693, dx=-0.0319, dy=-3.8178
2022-07-10 19:50:30,658 - stpipe.Spec2Pipeline.extract_1d - INFO - Extracting using transformation parameters [ 0.26929011 -0.0319002 -3.81779367]
2022-07-10 19:50:31,485 - stpipe.Spec2Pipeline.extract_1d - INFO - Solving for the optimal Tikhonov factor.
2022-07-10 19:50:33,236 - stpipe.Spec2Pipeline.extract_1d - INFO - Testing factors...
2022-07-10 19:50:33,759 - stpipe.Spec2Pipeline.extract_1d - INFO - 0/10
2022-07-10 19:50:34,279 - stpipe.Spec2Pipeline.extract_1d - INFO - 1/10
2022-07-10 19:50:34,805 - stpipe.Spec2Pipeline.extract_1d - INFO - 2/10
... ```

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Hi Aiden, I was looking into this but didn't have any errors when running the rate file through the pipeline. Do you have the association file on hand, and if so, does it have more than one file listed?

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Hi Tyler! If by association file you mean the jw01516-o005_20220710t191635_spec2_001_asn.json file, it only has one file listed inside, which is jw01516005001_03101_00001_nis_rate.fits

Here is the full contents of that .json file, if you're interested:

{
    "asn_type": "spec2",
    "asn_rule": "candidate_Asn_Lv2Spec",
    "version_id": "20220710t191635",
    "code_version": "1.5.3",
    "degraded_status": "No known degraded exposures in association.",
    "program": "01516",
    "constraints": "DMSAttrConstraint({'name': 'program', 'sources': ['program'], 'value': '1516'})\nDMSAttrConstraint({'name': 'is_tso', 'sources': ['tsovisit'], 'value': None})\nDMSAttrConstraint({'name': 'instrument', 'sources': ['instrume'], 'value': 'niriss'})\nDMSAttrConstraint({'name': 'detector', 'sources': ['detector'], 'value': 'nis'})\nDMSAttrConstraint({'name': 'opt_elem', 'sources': ['filter'], 'value': 'clear'})\nDMSAttrConstraint({'name': 'opt_elem2', 'sources': ['pupil'], 'value': 'gr700xd'})\nDMSAttrConstraint({'name': 'opt_elem3', 'sources': ['fxd_slit'], 'value': None})\nDMSAttrConstraint({'name': 'subarray', 'sources': ['subarray'], 'value': 'substrip256'})\nDMSAttrConstraint({'name': 'channel', 'sources': ['channel'], 'value': None})\nDMSAttrConstraint({'name': 'slit', 'sources': ['fxd_slit'], 'value': None})\nDMSAttrConstraint({'name': 'exp_type', 'sources': ['exp_type'], 'value': 'nis_soss'})\nConstraint_Single_Science({'name': 'single_science', 'value': False})\nDMSAttrConstraint({'name': 'patttype', 'sources': ['patttype'], 'value': ['2-point-nod|4-point-nod']})\nDMSAttrConstraint({'name': 'asn_candidate', 'sources': ['asn_candidate'], 'value': \"\\\\('o005',\\\\ 'observation'\\\\)\"})",
    "asn_id": "o005",
    "asn_pool": "jw01516_20220710t191635_pool.csv",
    "target": "4",
    "products": [
        {
            "name": "jw01516005001_03101_00001_nis",
            "members": [
                {
                    "expname": "jw01516005001_03101_00001_nis_rate.fits",
                    "exptype": "science",
                    "exposerr": "null"
                }
            ]
        }
    ] ```

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Aiden Kovacs the very top line of the error log that you included shows one of the reference files in use by the extract_1d step as: Reference type 'speckernel' defined as 'jwst_niriss_speckernel_0004.fits' Can you find the entries for the other 3 ref files used by the step, which are specprofile, spectrace, and wavemap? Need to verify which versions of those ref files were used. Thanks.

 

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Here are the entries for the other reference files (found in the same .err log), all are using jwst_0916.pmap :

2022-07-10 23:34:39,662 - CRDS - DEBUG - Reference type 'specprofile' defined as 'jwst_niriss_specprofile_0017.fits'

2022-07-10 23:34:39,643 - CRDS - DEBUG - Reference type 'spectrace' defined as 'jwst_niriss_spectrace_0018.fits'

2022-07-10 23:34:39,653 - CRDS - DEBUG - Reference type 'wavemap' defined as 'jwst_niriss_wavemap_0013.fits'

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Thanks Aiden Kovacs . This processing used old (pre-flight) ref files, because updated ref files that were delivered by the SOSS group recently have not yet been installed in CRDS OPS. And we're getting the impression (via a lot of testing) that in-flight SOSS data just don't process well using the pre-flight ref files. When we process this dataset using the in-flight ref files (which are on CRDS-PUB), it succeeds.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Problem was due to incompatibility between old (pre-flight) reference file data and in-flight science data. Fixed by using latest reference files, once they're in CRDS-OPS.

[stscijgbot-jp]

Comment by Jenny Shih on JIRA:

jw01538-o067_20220702t101353_tso-spec2_002 failed on 08/14/22 with what appeared to be the same error message. The context was jwst_0944.pmap and the following reference files were used

2022-08-14 12:02:19,878 - CRDS - DEBUG - Reference type 'speckernel' defined as 'jwst_niriss_speckernel_0004.fits' 2022-08-14 12:02:19,879 - CRDS - DEBUG - Reference type 'specprofile' defined as 'jwst_niriss_specprofile_0020.fits' 2022-08-14 12:02:19,879 - CRDS - DEBUG - Reference type 'spectrace' defined as 'jwst_niriss_spectrace_0022.fits' 2022-08-14 12:02:19,879 - CRDS - DEBUG - Reference type 'wavemap' defined as 'jwst_niriss_wavemap_0020.fits' 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Tested the jw01538-o067 dataset mentioned above using the latest B9dev version of the Cal pipeline, choosing the same set of reference files as above, and the processing succeeded. So this might get fixed once B8.1 is installed in Ops, or at the latest when B9 is installed.

[stscijgbot-jp]

Comment by John Scott on JIRA:

Similar error seen in jw01538-o067_20220702t101353_tso-spec2_002, failed again with build 8.1

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

We have to ask Loic Albert about this issue. He created the SOSS mode reference files that were submitted to CRDS at the end of commissioning. Nothing was done locally except to deliver the files to CRDS. There may be some issue in the ATOCA code version between what Loic used to make the reference files and whatever is currently being tested.

[stscijgbot-jp]

Comment by John Scott on JIRA:

Same level 2B error seen in reprocessing in build 8.1.2 for NIR_SOSS

jw01201-o002_20220922t140550_tso-spec2_024
jw01201-o002_20220922t140550_tso-spec2_026 jw01201-o002_20220922t140550_tso-spec2_021 jw01201-o002_20220922t140550_tso-spec2_006

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

John Scott Did the datasets mentioned above, which failed in B8.1.2, get reprocessed from scratch or did they just get retriggered starting at level-2b?  They may succeed if starting over from level-2a.

[stscijgbot-jp]

Comment by John Scott on JIRA:

Howard Bushouse these were reprocessed from level 05 (pod file)

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Mike Swam or John Scott Could you retrieve and attach to the ticket the most recent tso-spec2 ASN files that failed, because as usual we can't get failures from MAST:

jw01201-o002_20220922t140550_tso-spec2_024 jw01201-o002_20220922t140550_tso-spec2_026 jw01201-o002_20220922t140550_tso-spec2_021 jw01201-o002_20220922t140550_tso-spec2_006

Thanks.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Kenneth MacDonald I've retrieved the data files for jw01516-o005 (one of the failures) and put them in the data directory listed above. For testing, run:

strun calwebb_spec2 jw01516-o005_20220710t191635_spec2_001_asn.json```

[stscijgbot-jp]

Comment by Mike Swam on JIRA:

I copied the 4 asn tables here: ████████████████████████████

Let me know if you have trouble getting them.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Thanks Mike. I've now downloaded the data files for the 4 jw01201-o002 spec2 ASNs that apparently failed in Ops using B8.1.2. The spec2 ASN files and input rateints files are in the data directory listed above. Can be tested using:

strun calwebb_spec2 jw01201-o002_20220922t140550_tso-spec2_xxx_asn.json```
Warning: the rateints files are huge, because each one contains data for 190 integrations.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Kenneth MacDonald Couldn't resist running one of the jw01201 datasets myself and have found that if we just run the old rateints files (created by the previous version of Cal and retrieved from the archive) through the current calwebb_spec2 pipeline it succeeds. However, if I start over from the uncal (raw) file and rerun it through calwebb_detector1 to create a new rateints file, that then crashes with the error reported above. So it appears that something in the current level-2a (calwebb_detector1) processing is resulting in some kind of bad data that calwebb_spec2 doesn't like. The results from starting over on all processing are in the "reproc" subdirectory in the data directory.

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

another case jw01538-o067_20221004t132953_tso-spec2_002, which was reprocessed starting from level 05 with b8.1.2

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Another case occurred in extract_1d in tso3 pipeline: jw02589-o002_20221014t095829_tso3_001 , which was reprocessed from level 05 with B8.1.3

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Two more cases in reprocessing with B8.1.3: jw01538-o067_20221017t055317_tso-spec2_001 and jw01538-o066_20221017t055317_tso3_001

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

another case seen for nis_soss tso3 dataset jw01201-o101_20221028t144209_tso3_001 in extract_1d using b8.1.3

IndexError: index 5270 is out of bounds for axis 0 with size 5270 ```

[hbushouse]

Fixed by #6945

[stscijgbot-jp]

Comment by John Scott on JIRA:

Another case is seen for nis_soss tso3 dataset jw02589-o002_20221014t095829_tso3_001 in extract_1d using b8.1.3

IndexError: index 3640 is out of bounds for axis 0 with size 3640

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Another case from 1538, seen in reprocessing using B9.0 jw01538-o066_20221126t051249_tso3_00002

[stscijgbot-jp]

Comment by Hien Tran on JIRA:

same filesets for jw01201-o002 are failing l2b during repro with b9.0.2 as in DMSOPS-646 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Tyler Pauly Do we think these errors will be fixed once JP-2807 is released in B9.1?

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

(Almost) all data in the ticket data directory has been run without error on dev - I ran one of the 1201 associations without issue, but as they're 190 integrations, processing is quite slow. I've started a second association and will report if an issue arises.

I don't see any data for one of the cases listed above (the jw02589 error), so I'll try to find MAST entries and test.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

jw02589002001_04101_00001-seg002_nis_calints.fits was successfully produced, so I think all error cases mentioned above have been successfully tested. Note that one exposure in this program uses F277W, which the IDT extraction algorithm will not calibrate. This is coded as intended behavior and not a bug, but it will result in no x1dints output (for jw02589002001_04102_00001-seg001_nis_rateints.fits).

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

The ATOCA algorithm  is supposed to be able to extract the F277W spectrum, at least that was discussed in the original planning for the algorithm.  If it does not allow that currently this is an issue, although it is not of immediate concern for this ticket.  I have heard from Nestor Espinoza that the F277W/GR700XD exposures have turned out to not be as useful as we thought pre-launch so we may need to rethink the whole question of getting these exposures in F277W anyway.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

I made the note for DMS OPS folks, but I assumed this was known behavior amongst the instrument team. I would've assumed that instrument team testing would have made this behavior known and obvious - is that not the case?

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

The testing was all done at Montreal and I was not involved so I am not in the loop about that.  I thought I extracted an F277W spectrum with ATOCA for commissioning with help from Loic, but in any case this is not something to worry about now.  I will have to discuss this with Loic and see what the situation is.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Given that Tyler Pauly has shown that all failure cases reported here succeed when processed with the (soon to be released) B9.1 software, except for the known case of F277W data, I'm going to declare this as resolved and ready to be tested once B9.1 is available in Ops.

[stscijgbot-jp]

Comment by Aiden Kovacs on JIRA:

Howard Bushouse As of 9.1.2 we no longer see this error