Closed stscieisenhamer closed 5 years ago
Since there are no sources in any of these data, is this just a question of how the Image3 pipeline should fail? This seems like a good situation where we want to appropriately catch this error but at the same time, sometimes it might be meaningful and sometimes it wouldn't be meaningful.
cc: @larrybradley
If this is a valid situation: I.e. there is no data to do proper processing, then yes, this will need to be handled more gracefully. A similar situation is already handled, #2441. A discussion with SDP/Archive will need be had on how to downstream document the situation, but that is a separate issue.
Naive question: Does image registration/combination require identifiable sources in the data? Can it not proceed with just WCS info?
tweakreg
needs sources. Without it, it can do nothing. I.e. it should be skipped.
That said, you don't know if there are sources in the image until you look for them. So it should be handled gracefully in tweakreg
I think. I.e. look for sources. If there are none, throw up hands, set to 'SKIPPED', warn the user, and move on to the resampling with the existing, un-tweaked WCS.
Btw, what is EMTB data?
Engineering Mockup Test Bed?
Who's in charge of tweakreg
these days and could add a trap for too few sources? @larrybradley @jdavies-st @mcara Wouldn't this already be handled via the minobj
parameter?
I'll work on this one.
@larrybradley OK, so this crashes before reaching the "alignment" part of the code. The crash happens in the https://github.com/spacetelescope/jwst/blob/d7daa6c9cc60fc9870add2950ca06a424056db28/jwst/tweakreg/tweakreg_catalog.py#L59 because when no sources are found, then sources
is an empty table not initialized to contain the required columns. IMO this is a bug in the photutils
and I'll open an issue/PR there.
Meanwhile, hardening the code in tweakreg_catalog.py
results (as expected by design) in the following exception:
................
File "/.../jwst/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/.../jwst/jwst/tweakreg/tweakreg_step.py", line 133, in process
sigma=self.sigma
File "/.../jwst/jwst/tweakreg/imalign.py", line 185, in align
refcat = RefCatalog(ref_imcat.catalog, name=ref_imcat.name)
File "/.../jwst/jwst/tweakreg/wcsimage.py", line 1305, in __init__
self.catalog = catalog
File "/.../jwst/jwst/tweakreg/wcsimage.py", line 1336, in catalog
raise ValueError("Catalog must contain at least one source.")
ValueError: Catalog must contain at least one source.
Now, if you want tweakreg
to quietly quit - I can do that too.
FYI: just not having sources in one of the images does not necessarily means that tweakreg will not be able to align that image. For example, in ACS images, if there are no stars in WFC1 chip but there are stars in WFC2 chip - we can still align WFC1 because it is constrained to "move" together with WFC2. The same happens with JSWT (even though there is only one image model per chip) when images are grouped. What I want to say is that this kind of analysis is more complex than simple checking of the catalog length.
Closed with #2796 merging
Issue
Running
calwebb_image3
on some EMTB data, the following occursShow log...
``` $ strun cfgs/calwebb_image3.cfg jw00601-o012_image3_001_asn.json 2018-10-17 13:18:04,033 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created. 2018-10-17 13:18:04,035 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created. 2018-10-17 13:18:04,036 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created. 2018-10-17 13:18:04,038 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created. 2018-10-17 13:18:04,039 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created. 2018-10-17 13:18:04,040 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created. 2018-10-17 13:18:04,041 - stpipe - INFO - Hostname: inanna2.local.stsci.edu 2018-10-17 13:18:04,041 - stpipe - INFO - OS: Darwin 2018-10-17 13:18:04,082 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('jw00601-o012_image3_001_asn.json',). 2018-10-17 13:18:04,719 - stpipe.Image3Pipeline - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/asdf/block.py:708: RuntimeWarning: invalid value encountered in greater if (array > 2**52).any(): 2018-10-17 13:18:04,773 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00004_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:05,171 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:05,590 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00005_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:05,594 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:05,999 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00001_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,003 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:06,437 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00003_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,441 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:06,958 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00002_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,963 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:07,480 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00006_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:07,484 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:07,485 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ... 2018-10-17 13:18:10,668 - stpipe.Image3Pipeline - INFO - Aligning input images... 2018-10-17 13:18:10,727 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (,).
2018-10-17 13:18:10,821 - stpipe.Image3Pipeline.tweakreg - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/detection/findstars.py:926: AstropyUserWarning: Sources were found, but none pass the sharpness and roundness criteria.
'and roundness criteria.', AstropyUserWarning)
----------------------------------------------------------------------
ERROR RUNNING STEP 'Image3Pipeline':
'id'
Traceback (most recent call last):
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline
step.run(*positional)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_image3.py", line 70, in process
input_models = self.tweakreg(input_models)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_step.py", line 69, in process
self.snr_threshold)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_catalog.py", line 59, in make_tweakreg_catalog
catalog = sources[columns]
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in __getitem__
out = self.__class__([self[x] for x in item],
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in
out = self.__class__([self[x] for x in item],
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1223, in __getitem__
return self.columns[item]
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 107, in __getitem__
return OrderedDict.__getitem__(self, item)
KeyError: 'id'
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/EGG-INFO/scripts/strun", line 26, in
step = Step.from_cmdline(sys.argv[1:])
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 162, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline
step.run(*positional)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_image3.py", line 70, in process
input_models = self.tweakreg(input_models)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run
step_result = self.process(*args)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_step.py", line 69, in process
self.snr_threshold)
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_catalog.py", line 59, in make_tweakreg_catalog
catalog = sources[columns]
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in __getitem__
out = self.__class__([self[x] for x in item],
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in
out = self.__class__([self[x] for x in item],
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1223, in __getitem__
return self.columns[item]
File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 107, in __getitem__
return OrderedDict.__getitem__(self, item)
KeyError: 'id'
```
Resources
Data can be found in
/grp/jwst/ssb/eisenhamer/jwst_issues/issue2520/analysis