search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
565 stars 167 forks source link

tweakreg: KeyError: 'id' #2716

Closed stscieisenhamer closed 5 years ago

stscieisenhamer commented 6 years ago

Issue

Running calwebb_image3 on some EMTB data, the following occurs

Show log...

``` $ strun cfgs/calwebb_image3.cfg jw00601-o012_image3_001_asn.json 2018-10-17 13:18:04,033 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created. 2018-10-17 13:18:04,035 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created. 2018-10-17 13:18:04,036 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created. 2018-10-17 13:18:04,038 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created. 2018-10-17 13:18:04,039 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created. 2018-10-17 13:18:04,040 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created. 2018-10-17 13:18:04,041 - stpipe - INFO - Hostname: inanna2.local.stsci.edu 2018-10-17 13:18:04,041 - stpipe - INFO - OS: Darwin 2018-10-17 13:18:04,082 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('jw00601-o012_image3_001_asn.json',). 2018-10-17 13:18:04,719 - stpipe.Image3Pipeline - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/asdf/block.py:708: RuntimeWarning: invalid value encountered in greater if (array > 2**52).any(): 2018-10-17 13:18:04,773 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00004_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:05,171 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:05,590 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00005_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:05,594 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:05,999 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00001_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,003 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:06,437 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00003_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,441 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:06,958 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00002_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:06,963 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:07,480 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00601012001_02102_00006_nrcblong_cal.fits' reftypes = ['drizpars'] 2018-10-17 13:18:07,484 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Downloads/crds-cache/references/jwst/nircam/jwst_nircam_drizpars_0001.fits'. 2018-10-17 13:18:07,485 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ... 2018-10-17 13:18:10,668 - stpipe.Image3Pipeline - INFO - Aligning input images... 2018-10-17 13:18:10,727 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (,). 2018-10-17 13:18:10,821 - stpipe.Image3Pipeline.tweakreg - WARNING - /Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/photutils/detection/findstars.py:926: AstropyUserWarning: Sources were found, but none pass the sharpness and roundness criteria. 'and roundness criteria.', AstropyUserWarning) ---------------------------------------------------------------------- ERROR RUNNING STEP 'Image3Pipeline': 'id' Traceback (most recent call last): File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline step.run(*positional) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_image3.py", line 70, in process input_models = self.tweakreg(input_models) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_step.py", line 69, in process self.snr_threshold) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_catalog.py", line 59, in make_tweakreg_catalog catalog = sources[columns] File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in __getitem__ out = self.__class__([self[x] for x in item], File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in out = self.__class__([self[x] for x in item], File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1223, in __getitem__ return self.columns[item] File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 107, in __getitem__ return OrderedDict.__getitem__(self, item) KeyError: 'id' ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/EGG-INFO/scripts/strun", line 26, in step = Step.from_cmdline(sys.argv[1:]) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 162, in from_cmdline return cmdline.step_from_cmdline(args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline step.run(*positional) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_image3.py", line 70, in process input_models = self.tweakreg(input_models) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 400, in run step_result = self.process(*args) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_step.py", line 69, in process self.snr_threshold) File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/jwst-0.13.0a0.dev23+gb3883581-py3.6-macosx-10.7-x86_64.egg/jwst/tweakreg/tweakreg_catalog.py", line 59, in make_tweakreg_catalog catalog = sources[columns] File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in __getitem__ out = self.__class__([self[x] for x in item], File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1229, in out = self.__class__([self[x] for x in item], File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 1223, in __getitem__ return self.columns[item] File "/Users/eisenham/anaconda3/envs/condadev/lib/python3.6/site-packages/astropy/table/table.py", line 107, in __getitem__ return OrderedDict.__getitem__(self, item) KeyError: 'id' ```

Resources

Data can be found in /grp/jwst/ssb/eisenhamer/jwst_issues/issue2520/analysis

stscicrawford commented 6 years ago

Since there are no sources in any of these data, is this just a question of how the Image3 pipeline should fail? This seems like a good situation where we want to appropriately catch this error but at the same time, sometimes it might be meaningful and sometimes it wouldn't be meaningful.

cc: @larrybradley

stscieisenhamer commented 6 years ago

If this is a valid situation: I.e. there is no data to do proper processing, then yes, this will need to be handled more gracefully. A similar situation is already handled, #2441. A discussion with SDP/Archive will need be had on how to downstream document the situation, but that is a separate issue.

Naive question: Does image registration/combination require identifiable sources in the data? Can it not proceed with just WCS info?

jdavies-st commented 6 years ago

tweakreg needs sources. Without it, it can do nothing. I.e. it should be skipped.

That said, you don't know if there are sources in the image until you look for them. So it should be handled gracefully in tweakreg I think. I.e. look for sources. If there are none, throw up hands, set to 'SKIPPED', warn the user, and move on to the resampling with the existing, un-tweaked WCS.

jdavies-st commented 6 years ago

Btw, what is EMTB data?

hbushouse commented 6 years ago

Engineering Mockup Test Bed?

hbushouse commented 6 years ago

Who's in charge of tweakreg these days and could add a trap for too few sources? @larrybradley @jdavies-st @mcara Wouldn't this already be handled via the minobj parameter?

mcara commented 6 years ago

I'll work on this one.

mcara commented 6 years ago

@larrybradley OK, so this crashes before reaching the "alignment" part of the code. The crash happens in the https://github.com/spacetelescope/jwst/blob/d7daa6c9cc60fc9870add2950ca06a424056db28/jwst/tweakreg/tweakreg_catalog.py#L59 because when no sources are found, then sources is an empty table not initialized to contain the required columns. IMO this is a bug in the photutils and I'll open an issue/PR there.

Meanwhile, hardening the code in tweakreg_catalog.py results (as expected by design) in the following exception:

................
  File "/.../jwst/jwst/stpipe/step.py", line 400, in run
    step_result = self.process(*args)
  File "/.../jwst/jwst/tweakreg/tweakreg_step.py", line 133, in process
    sigma=self.sigma
  File "/.../jwst/jwst/tweakreg/imalign.py", line 185, in align
    refcat = RefCatalog(ref_imcat.catalog, name=ref_imcat.name)
  File "/.../jwst/jwst/tweakreg/wcsimage.py", line 1305, in __init__
    self.catalog = catalog
  File "/.../jwst/jwst/tweakreg/wcsimage.py", line 1336, in catalog
    raise ValueError("Catalog must contain at least one source.")
ValueError: Catalog must contain at least one source.

Now, if you want tweakreg to quietly quit - I can do that too.

FYI: just not having sources in one of the images does not necessarily means that tweakreg will not be able to align that image. For example, in ACS images, if there are no stars in WFC1 chip but there are stars in WFC2 chip - we can still align WFC1 because it is constrained to "move" together with WFC2. The same happens with JSWT (even though there is only one image model per chip) when images are grouped. What I want to say is that this kind of analysis is more complex than simple checking of the catalog length.

stscicrawford commented 5 years ago

Closed with #2796 merging