search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

spec2 crashing on nis_soss jw04098-o007_20230923t230916_tso-spec2_00003 #8110

Closed stscijgbot-jp closed 9 months ago

stscijgbot-jp commented 11 months ago

Issue JP-3483 was created on JIRA by Hien Tran:

we have a L2B failure for new NIRISS TSO dataset (not repro) executed 2023nov10, jw04098-o007_20230923t230916_tso-spec2_00003, with the error message


dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=0 ```
the full traceback, which can be found at <https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw04098-o007_20230923t230916_tso-spec2_00003_1699631042.752258&fileset_name=jw04098-o007_20230923t230916_tso-spec2_00003&dag_name=LEVEL_2B> 
or
*Directory Path:* /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw04098-o007_20230923t230916_tso-spec2_00003_1699631042.752258,
is:
```java
2023-11-10 20:13:14,075 - CRDS - DEBUG - Final effective context is 'jwst_1147.pmap'
2023-11-10 20:13:14,075 - CRDS - DEBUG - Computing best references locally.
2023-11-10 20:13:14,076 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NISRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NIS_SOSS',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NIS',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRISS',
'META.INSTRUMENT.PUPIL [PUPIL]': 'GR700XD',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-11-10',
'META.OBSERVATION.TIME [TIME-OBS]': '08:26:26.115',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBSTRIP256',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2023-11-10 20:13:14,076 - CRDS - DEBUG - Reference type 'speckernel' defined as 'jwst_niriss_speckernel_0004.fits'
2023-11-10 20:13:14,565 - stpipe.Spec2Pipeline.extract_1d - INFO - Wavelength grid was not specified. Setting `wave_grid` to None.
2023-11-10 20:13:14,637 - stpipe.Spec2Pipeline.extract_1d - INFO - Input is a CubeModel containing 188 integrations.
2023-11-10 20:13:14,637 - stpipe.Spec2Pipeline.extract_1d - INFO - Processing integration 1 of 188.
2023-11-10 20:13:14,642 - stpipe.Spec2Pipeline.extract_1d - INFO - Skip background subtraction.
2023-11-10 20:13:14,643 - stpipe.Spec2Pipeline.extract_1d - INFO - Solving for the transformation parameters.
2023-11-10 20:13:14,643 - stpipe.Spec2Pipeline.extract_1d - INFO - Parameters used for fit: theta, x-offset, y-offset
2023-11-10 20:13:14,643 - stpipe.Spec2Pipeline.extract_1d - INFO - Fixed parameters:
2023-11-10 20:13:14,916 - stpipe.Spec2Pipeline.extract_1d - INFO - Measuring trace position for orders 1 and 2.
2023-11-10 20:13:15,223 - stpipe.Spec2Pipeline.extract_1d - INFO - Measured to Reference trace position transform: theta=-0.00038201755227297866, x-offset=-0.24237455427848956, y-offset=0.5404013401742825
2023-11-10 20:13:17,452 - stpipe.Spec2Pipeline.extract_1d - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/scipy/sparse/linalg/_dsolve/linsolve.py:230: MatrixRankWarning: Matrix is exactly singular
warn("Matrix is exactly singular", MatrixRankWarning)
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 122, in process
result = self.process_exposure_product(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 325, in process_exposure_product
x1d = self.extract_1d(x1d)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/extract_1d_step.py", line 416, in process
result, ref_outputs, atoca_outputs = soss_extract.run_extract1d(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 1183, in run_extract1d
result = model_image(scidata_bkg, scierr, scimask, refmask, ref_files, box_weights, subarray, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 556, in model_image
estimate = estim_flux_first_order(scidata_bkg, scierr, scimask,
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 202, in estim_flux_first_order
estimate_spl = UnivariateSpline(wave_grid[idx], spec_estimate[idx], k=3, s=0, ext=0)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/scipy/interpolate/_fitpack2.py", line 203, in __init__
data = dfitpack.fpcurf0(x, y, k, w=w, xb=bbox[0],
dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=0
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/bin/strun", line 26, in <module>
step = Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 186, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/cmdline.py", line 386, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 140, in process
raise RuntimeError('\n'.join(failures))
RuntimeError: Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 122, in process
result = self.process_exposure_product(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 325, in process_exposure_product
x1d = self.extract_1d(x1d)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/extract_1d_step.py", line 416, in process
result, ref_outputs, atoca_outputs = soss_extract.run_extract1d(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 1183, in run_extract1d
result = model_image(scidata_bkg, scierr, scimask, refmask, ref_files, box_weights, subarray, **kwargs)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 556, in model_image
estimate = estim_flux_first_order(scidata_bkg, scierr, scimask,
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 202, in estim_flux_first_order
estimate_spl = UnivariateSpline(wave_grid[idx], spec_estimate[idx], k=3, s=0, ext=0)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/scipy/interpolate/_fitpack2.py", line 203, in __init__
data = dfitpack.fpcurf0(x, y, k, w=w, xb=bbox[0],
dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=0
2023314201318 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw04098-o007_20230923t230916_tso-spec2_00003_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.11.4.20230814-py3.9/lib/python3.9/site-packages/jwst/pipeline/calwebb_tso-spec2.cfg FAILED (exit=1) on jw04098-o007_20230923t230916_tso-spec2_00003_asn.json."```
the failure occurred for only this **third** segment (`{}jw04098007001_04101_00001-{*}seg003{*}_nis){`} out of 5 segments for this nis_soss exposure. the other four segments went through just fine. may or may not be related to the context update 1147
stscijgbot-jp commented 11 months ago

Comment by Hien Tran on JIRA:

a new case seen for jw04098-o012_20231206t214017_tso-spec2_00001 executed 2023-12-06. same error message, under b10.0 (jwstdp-1.12.5) with context jwst_1166


2023-12-06 21:52:37,046 - CRDS - DEBUG - Final effective context is 'jwst_1166.pmap'
2023-12-06 21:52:37,046 - CRDS - DEBUG - Computing best references locally.
2023-12-06 21:52:37,046 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NISRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NIS_SOSS',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NIS',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRISS',
'META.INSTRUMENT.PUPIL [PUPIL]': 'GR700XD',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-12-06',
'META.OBSERVATION.TIME [TIME-OBS]': '06:00:03.152',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUBSTRIP256',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'T',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2023-12-06 21:52:37,047 - CRDS - DEBUG - Reference type 'speckernel' defined as 'jwst_niriss_speckernel_0004.fits'
2023-12-06 21:52:37,409 - stpipe.Spec2Pipeline.extract_1d - INFO - Wavelength grid was not specified. Setting `wave_grid` to None.
2023-12-06 21:52:37,457 - stpipe.Spec2Pipeline.extract_1d - INFO - Input is a CubeModel containing 19 integrations.
2023-12-06 21:52:37,457 - stpipe.Spec2Pipeline.extract_1d - INFO - Processing integration 1 of 19.
2023-12-06 21:52:37,461 - stpipe.Spec2Pipeline.extract_1d - INFO - Skip background subtraction.
2023-12-06 21:52:37,462 - stpipe.Spec2Pipeline.extract_1d - INFO - Solving for the transformation parameters.
2023-12-06 21:52:37,462 - stpipe.Spec2Pipeline.extract_1d - INFO - Parameters used for fit: theta, x-offset, y-offset
2023-12-06 21:52:37,462 - stpipe.Spec2Pipeline.extract_1d - INFO - Fixed parameters:
2023-12-06 21:52:37,671 - stpipe.Spec2Pipeline.extract_1d - INFO - Measuring trace position for orders 1 and 2.
2023-12-06 21:52:37,916 - stpipe.Spec2Pipeline.extract_1d - INFO - Measured to Reference trace position transform: theta=-0.0004479779977205849, x-offset=-0.13957894378664792, y-offset=0.5751044504934412
2023-12-06 21:52:39,695 - stpipe.Spec2Pipeline.extract_1d - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/scipy/sparse/linalg/_dsolve/linsolve.py:230: MatrixRankWarning: Matrix is exactly singular
2023-12-06 21:52:39,695 - stpipe.Spec2Pipeline.extract_1d - WARNING - warn("Matrix is exactly singular", MatrixRankWarning)
2023-12-06 21:52:39,695 - stpipe.Spec2Pipeline.extract_1d - WARNING -
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_spec2.py", line 122, in process
result = self.process_exposure_product(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_spec2.py", line 322, in process_exposure_product
x1d = self.extract_1d(x1d)
^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/extract_1d_step.py", line 456, in process
result, ref_outputs, atoca_outputs = soss_extract.run_extract1d(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 1183, in run_extract1d
result = model_image(scidata_bkg, scierr, scimask, refmask, ref_files, box_weights, subarray, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 556, in model_image
estimate = estim_flux_first_order(scidata_bkg, scierr, scimask,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 202, in estim_flux_first_order
estimate_spl = UnivariateSpline(wave_grid[idx], spec_estimate[idx], k=3, s=0, ext=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/scipy/interpolate/_fitpack2.py", line 203, in __init__
data = dfitpack.fpcurf0(x, y, k, w=w, xb=bbox[0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=0
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/bin/strun", line 26, in <module>
step = Step.from_cmdline(sys.argv[1:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 186, in from_cmdline
return cmdline.step_from_cmdline(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/cmdline.py", line 386, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_spec2.py", line 140, in process
raise RuntimeError('\n'.join(failures))
RuntimeError: Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_spec2.py", line 122, in process
result = self.process_exposure_product(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_spec2.py", line 322, in process_exposure_product
x1d = self.extract_1d(x1d)
^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/extract_1d_step.py", line 456, in process
result, ref_outputs, atoca_outputs = soss_extract.run_extract1d(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 1183, in run_extract1d
result = model_image(scidata_bkg, scierr, scimask, refmask, ref_files, box_weights, subarray, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 556, in model_image
estimate = estim_flux_first_order(scidata_bkg, scierr, scimask,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/extract_1d/soss_extract/soss_extract.py", line 202, in estim_flux_first_order
estimate_spl = UnivariateSpline(wave_grid[idx], spec_estimate[idx], k=3, s=0, ext=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/scipy/interpolate/_fitpack2.py", line 203, in __init__
data = dfitpack.fpcurf0(x, y, k, w=w, xb=bbox[0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=0 ```
stscijgbot-jp commented 10 months ago

Comment by Howard Bushouse on JIRA:

Hien Tran I grabbed jw04098007001_04101_00001-seg003_nis_rateints.fits from MAST and am able to successfully process it using the latest b10.1 version of the pipeline and the latest crds context. But regarding CRDS ref files, it looks to me like I'm using all the same ones that were used in operations, because none of the SOSS-specific ones have had any updates in a while. And the latest photom ref file for SOSS mode should be irrelevant here, because the photom step is applied after extract_1d for SOSS data. So the crash in Ops hadn't even gotten to photom yet. So I'm not sure if something in either b10.0 or b10.1 has fixed this problem or maybe it just runs OK under Mac OS but not Linux. It's worth trying again under b10.0 and certainly again under b10.1 when it gets to Ops in about a month.

stscijgbot-jp commented 10 months ago

Comment by Howard Bushouse on JIRA:

I have confirmed that jw04098012001_04101_00001-seg005_nis_rateints.fits crashes for me too, with the same error signature, when attempting to process using the latest jwst pipeline code. This is NOT a full-frame image; it uses the SUBSTRIP256 subarray, which is quite standard for SOSS. So this has got to be some kind of mathematical error within the ATOCA algorithm.

stscijgbot-jp commented 9 months ago

Comment by Ned Molter on JIRA:

This is happening because the approximations used by scipy.sparse.linalg.spsolve occasionally find zero solutions despite perfectly reasonable-looking inputs, raising a MatrixRankWarning and then failing in the next step.  Apparently, the approximation of the matrix has determinant 0 when it shouldn't, but I'm definitely no linear algebra expert.  In my proposed fix, I am catching the MatrixRankWarning, and if present, reverting to a different solver (scipy.sparse.linalg.lsqr) which is probably slower but more stable.  The data and input parameters that were causing the failure pass through ok, and the output spectra look qualitatively reasonable, so I'm planning to proceed with that fix.

stscijgbot-jp commented 9 months ago

Comment by Howard Bushouse on JIRA:

Fixed by #8273

stscijgbot-jp commented 6 months ago

Comment by Joseph Filippazzo on JIRA:

I ran the dataset /ifs/jwst/wit/niriss/soss/uncal_data/jw04098012001/jw04098012001_04101_00001-seg005_nis_uncal.fits through the DETECTOR1 and SPEC2 pipelines with jwst v1.14.0 and it worked with no problems. The Jupyter notebook I made for the test is at /ifs/jwst/wit/niriss/pipeline_testing/build_10.2/JP-3483/JP-3483 Testing.ipynb.