search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
569 stars 167 forks source link

skipped tweakwcs step during niriss image3 got restarted anyway #8435

Closed stscijgbot-jp closed 6 months ago

stscijgbot-jp commented 6 months ago

Issue JP-3604 was created on JIRA by Hien Tran:

during reprocessing of nis_image dataset jw01537-o003_20240406t164421_image3_00004 with b10.1.2 ops calibration pipeline crashed when tweakwcs in the tweakreg step correctly got skipped, but it immediately started up again, which then crashed: 


2024-04-18 12:27:08,503 - CRDS - DEBUG - Final effective context is 'jwst_1225.pmap'
2024-04-18 12:27:08,503 - CRDS - DEBUG - Computing best references locally.
2024-04-18 12:27:08,504 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NISRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NIS_IMAGE',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NIS',
'META.INSTRUMENT.FILTER [FILTER]': 'CLEAR',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRISS',
'META.INSTRUMENT.PUPIL [PUPIL]': 'F158M',
'META.OBSERVATION.DATE [DATE-OBS]': '2022-09-21',
'META.OBSERVATION.TIME [TIME-OBS]': '09:44:37.856',
'META.SUBARRAY.NAME [SUBARRAY]': 'WFSS64R',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'F',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2024-04-18 12:27:08,505 - CRDS - DEBUG - Reference type 'abvegaoffset' defined as 'jwst_niriss_abvegaoffset_0003.asdf'
2024-04-18 12:27:08,505 - CRDS - DEBUG - Reference type 'apcorr' defined as 'jwst_niriss_apcorr_0008.fits'
2024-04-18 12:27:08,505 - CRDS - DEBUG - Reference type 'drizpars' defined as 'jwst_niriss_drizpars_0002.fits'
2024-04-18 12:27:08,511 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_abvegaoffset_0003.asdf'.
2024-04-18 12:27:08,513 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_apcorr_0008.fits'.
2024-04-18 12:27:08,513 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_drizpars_0002.fits'.
2024-04-18 12:27:08,515 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2024-04-18 12:27:09,228 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2024-04-18 12:27:09,229 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'save_catalogs': False, 'use_custom_catalogs': False, 'catalog_format': 'ecsv', 'catfile': '', 'kernel_fwhm': 2.0, 'snr_threshold': 10, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 100, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 1.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 1.0, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3.0, 'abs_refcat': 'GAIADR3', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}
2024-04-18 12:27:09,238 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:273: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-04-18 12:27:09,238 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-04-18 12:27:09,238 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,292 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/detection/daofinder.py:226: NoDetectionsWarning: No sources were found.
2024-04-18 12:27:09,292 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('No sources were found.', NoDetectionsWarning)
2024-04-18 12:27:09,292 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,294 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in jw01537003001_02109_00001_nis_cal.fits.
2024-04-18 12:27:09,305 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:273: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-04-18 12:27:09,305 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-04-18 12:27:09,305 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,356 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/detection/daofinder.py:226: NoDetectionsWarning: No sources were found.
2024-04-18 12:27:09,356 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('No sources were found.', NoDetectionsWarning)
2024-04-18 12:27:09,356 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,358 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in jw01537003001_02109_00002_nis_cal.fits.
2024-04-18 12:27:09,368 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:273: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-04-18 12:27:09,368 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-04-18 12:27:09,368 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,421 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/detection/daofinder.py:226: NoDetectionsWarning: No sources were found.
2024-04-18 12:27:09,421 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('No sources were found.', NoDetectionsWarning)
2024-04-18 12:27:09,421 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,422 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in jw01537003001_02109_00003_nis_cal.fits.
2024-04-18 12:27:09,432 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:273: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-04-18 12:27:09,432 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-04-18 12:27:09,432 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,484 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/photutils/detection/daofinder.py:226: NoDetectionsWarning: No sources were found.
2024-04-18 12:27:09,485 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('No sources were found.', NoDetectionsWarning)
2024-04-18 12:27:09,485 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-04-18 12:27:09,486 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in jw01537003001_02109_00004_nis_cal.fits.
2024-04-18 12:27:09,495 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:09,495 - stpipe.Image3Pipeline.tweakreg - INFO - Number of image groups to be aligned: 4.
2024-04-18 12:27:09,495 - stpipe.Image3Pipeline.tweakreg - INFO - Image groups:
2024-04-18 12:27:09,516 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'jw01537003001_02109_00001_nis_cal':
2024-04-18 12:27:09,516 - stpipe.Image3Pipeline.tweakreg - INFO - jw01537003001_02109_00001_nis_cal
2024-04-18 12:27:09,538 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'jw01537003001_02109_00002_nis_cal':
2024-04-18 12:27:09,538 - stpipe.Image3Pipeline.tweakreg - INFO - jw01537003001_02109_00002_nis_cal
2024-04-18 12:27:09,559 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'jw01537003001_02109_00003_nis_cal':
2024-04-18 12:27:09,559 - stpipe.Image3Pipeline.tweakreg - INFO - jw01537003001_02109_00003_nis_cal
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'jw01537003001_02109_00004_nis_cal':
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO - jw01537003001_02109_00004_nis_cal
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() started on 2024-04-18 12:27:09.580802
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO - Version 0.8.5
2024-04-18 12:27:09,580 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:09,629 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'jw01537003001_02109_00001_nis_cal' will not be aligned: empty source catalog
2024-04-18 12:27:09,659 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'jw01537003001_02109_00002_nis_cal' will not be aligned: empty source catalog
2024-04-18 12:27:09,689 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'jw01537003001_02109_00003_nis_cal' will not be aligned: empty source catalog
2024-04-18 12:27:09,719 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'jw01537003001_02109_00004_nis_cal' will not be aligned: empty source catalog
2024-04-18 12:27:09,719 - stpipe.Image3Pipeline.tweakreg - WARNING - Too few input images (or groups of images) with non-empty catalogs.
2024-04-18 12:27:09,719 - stpipe.Image3Pipeline.tweakreg - WARNING - At least two exposures are required for image alignment.
2024-04-18 12:27:09,719 - stpipe.Image3Pipeline.tweakreg - WARNING - Nothing to do. Skipping 'TweakRegStep'...
2024-04-18 12:27:10,120 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:10,120 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() started on 2024-04-18 12:27:10.120583
2024-04-18 12:27:10,120 - stpipe.Image3Pipeline.tweakreg - INFO - Version 0.8.5
2024-04-18 12:27:10,120 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-04-18 12:27:11,090 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID '987654' will not be aligned: empty source catalog
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/stpipe/cli/strun.py", line 24, in main
Step.from_cmdline(sys.argv[1:])
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 187, in from_cmdline
return cmdline.step_from_cmdline(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/stpipe/cmdline.py", line 392, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 479, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_image3.py", line 89, in process
input_models = self.tweakreg(input_models)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 479, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/jwst/tweakreg/tweakreg_step.py", line 442, in process
align_wcs(
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/tweakwcs/imalign.py", line 652, in align_wcs
raise ValueError("Too few input images (or groups of images) with "
ValueError: Too few input images (or groups of images) with non-empty catalogs.
2024109122711 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw01537-o003_20240406t164421_image3_00004_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.13.3.20240108-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_image3.cfg FAILED (exit=1) on jw01537-o003_20240406t164421_image3_00004_asn.json."
2024109122711 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw01537-o003_20240406t164421_image3_00004_asn msg="strun exit status=1"```
the issue appeared to have started in dms b10.0, before which reprocessing of the same data showed tweakreg being skipped without erroring. 
stscijgbot-jp commented 6 months ago

Comment by Howard Bushouse on JIRA:

Hien Tran We have a solution for this that we'd like to test using the same input data as was used in Ops. I could get files from MAST, but we'd really rather have the identical inputs as used most recently in Ops. Since I don't have access to the Ops server, could you please copy the image3 asn file and all input cal files (listed in the asn file) to someplace that we can get to them? Thanks.  And same request for the data involved in JP-3605.

stscijgbot-jp commented 6 months ago

Comment by Hien Tran on JIRA:

Howard Bushouse the data, which is public, can be found in ████████████████████████████

stscijgbot-jp commented 6 months ago

Comment by Howard Bushouse on JIRA:

Thanks Hien Tran for the data files. We've tested the proposed fix and it runs fine.

stscijgbot-jp commented 6 months ago

Comment by Howard Bushouse on JIRA:

Fixed by #8450

stscijgbot-jp commented 1 month ago

Comment by David Law on JIRA:

Hien Tran How's this looking in ops for 11.0?

stscijgbot-jp commented 1 week ago

Comment by Hien Tran on JIRA:

this dataset ran through level3 successfully during repro on ops with b11.0.2 on 2024oct04


jw01537-o003_20241003t130922_image3_00004          REPRO_LEVEL_3        2024-10-04 00:32:03  2024-10-04 00:32:27  Exited          0          owlmgr            /ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01537-o003_20241003t130922_image3_00004_1728001233.699478  ```
closing.