search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

Outlier detection blot for MIRI Fixed slit fails due to NaNs in pixmap #2473

Closed jdavies-st closed 5 years ago

jdavies-st commented 6 years ago

Working with recent SDP data resolving another issue, #2395, a new error has cropped up. Data is here:

/grp/jwst/ssb/eisenhamer/i2338

And the error occurs when outlier_detection blots back the median image to each input frame. But for MIRI fixed slit, the input frame is the full detector. Meanwhile, the median is just the spectrum inside the bounding box resampled. The median knows nothing about pixels outside the bounding box. So the call to drizzle.tblot inludes NaNs in the pixmap, which it cannot deal with because blot uses interpolation.

2018-09-06 12:27:54,460 - stpipe.Spec3Pipeline - INFO - Spec3Pipeline instance created.
2018-09-06 12:27:54,462 - stpipe.Spec3Pipeline.mrs_imatch - INFO - MRSIMatchStep instance created.
2018-09-06 12:27:54,464 - stpipe.Spec3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2018-09-06 12:27:54,466 - stpipe.Spec3Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2018-09-06 12:27:54,469 - stpipe.Spec3Pipeline.cube_build - INFO - CubeBuildStep instance created.
2018-09-06 12:27:54,470 - stpipe.Spec3Pipeline.extract_1d - INFO - Extract1dStep instance created.
2018-09-06 12:27:54,474 - stpipe - INFO - Hostname: rey.stsci.edu
2018-09-06 12:27:54,474 - stpipe - INFO - OS: Darwin
2018-09-06 12:27:54,540 - stpipe.Spec3Pipeline - INFO - Step Spec3Pipeline running with args ('output/jw98010-o001_20180806t184118_spec3_001_asn.json',).
2018-09-06 12:27:54,761 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw98010001001_02104_00001_mirimage_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol']
2018-09-06 12:27:54,840 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'.
2018-09-06 12:27:54,841 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2018-09-06 12:27:54,846 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/grp/crds/cache/references/jwst/jwst_miri_extract1d_0004.json'.
2018-09-06 12:27:54,850 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'.
2018-09-06 12:27:55,035 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw98010001001_02103_00001_mirimage_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol']
2018-09-06 12:27:55,038 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'.
2018-09-06 12:27:55,038 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2018-09-06 12:27:55,041 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/grp/crds/cache/references/jwst/jwst_miri_extract1d_0004.json'.
2018-09-06 12:27:55,045 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'.
2018-09-06 12:27:55,229 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw98010001001_02101_00001_mirimage_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol']
2018-09-06 12:27:55,232 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'.
2018-09-06 12:27:55,233 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2018-09-06 12:27:55,236 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/grp/crds/cache/references/jwst/jwst_miri_extract1d_0004.json'.
2018-09-06 12:27:55,241 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'.
2018-09-06 12:27:55,419 - stpipe.Spec3Pipeline - INFO - Prefetching reference files for dataset: 'jw98010001001_02102_00001_mirimage_cal.fits' reftypes = ['cubepar', 'drizpars', 'extract1d', 'resol']
2018-09-06 12:27:55,423 - stpipe.Spec3Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'.
2018-09-06 12:27:55,423 - stpipe.Spec3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2018-09-06 12:27:55,426 - stpipe.Spec3Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/grp/crds/cache/references/jwst/jwst_miri_extract1d_0004.json'.
2018-09-06 12:27:55,428 - stpipe.Spec3Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'.
2018-09-06 12:27:55,428 - stpipe.Spec3Pipeline - INFO - Starting calwebb_spec3 ...
2018-09-06 12:27:55,689 - stpipe.Spec3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2018-09-06 12:27:55,692 - stpipe.Spec3Pipeline.outlier_detection - INFO - Performing outlier detection on 4 inputs
2018-09-06 12:27:55,732 - stpipe.Spec3Pipeline.outlier_detection - WARNING - /Users/jdavies/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/numpy/lib/function_base.py:3250: RuntimeWarning: All-NaN slice encountered
  r = func(a, **kwargs)

2018-09-06 12:27:56,372 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 43)
2018-09-06 12:27:57,048 - stpipe.Spec3Pipeline.outlier_detection - WARNING - /Users/jdavies/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/scipy/interpolate/interpolate.py:2532: RuntimeWarning: invalid value encountered in less
  out_of_bounds += x < grid[0]

2018-09-06 12:27:57,049 - stpipe.Spec3Pipeline.outlier_detection - WARNING - /Users/jdavies/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/scipy/interpolate/interpolate.py:2533: RuntimeWarning: invalid value encountered in greater
  out_of_bounds += x > grid[-1]

2018-09-06 12:27:57,094 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 43)
2018-09-06 12:27:57,150 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 43)
2018-09-06 12:27:57,816 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 43)
2018-09-06 12:27:57,874 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 43)
2018-09-06 12:27:58,561 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 43)
2018-09-06 12:27:58,617 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 43)
2018-09-06 12:27:59,319 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 43)
2018-09-06 12:27:59,380 - stpipe.Spec3Pipeline.outlier_detection - INFO - Blotting median...
2018-09-06 12:28:00,088 - stpipe.Spec3Pipeline.outlier_detection - WARNING - /Users/jdavies/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/scipy/interpolate/interpolate.py:2532: RuntimeWarning: invalid value encountered in less
  out_of_bounds += x < grid[0]

2018-09-06 12:28:00,090 - stpipe.Spec3Pipeline.outlier_detection - WARNING - /Users/jdavies/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/scipy/interpolate/interpolate.py:2533: RuntimeWarning: invalid value encountered in greater
  out_of_bounds += x > grid[-1]

2018-09-06 12:28:00,132 - stpipe.Spec3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (387, 43)
----------------------------------------------------------------------
ERROR RUNNING STEP 'Spec3Pipeline':
    NaN in pixmap[0,0]
Traceback (most recent call last):
  File "/Users/jdavies/dev/jwst/jwst/stpipe/cmdline.py", line 297, in step_from_cmdline
    step.run(*positional)
  File "/Users/jdavies/dev/jwst/jwst/stpipe/step.py", line 400, in run
    step_result = self.process(*args)
  File "/Users/jdavies/dev/jwst/jwst/pipeline/calwebb_spec3.py", line 120, in process
    result = self.outlier_detection(result)
  File "/Users/jdavies/dev/jwst/jwst/stpipe/step.py", line 400, in run
    step_result = self.process(*args)
  File "/Users/jdavies/dev/jwst/jwst/outlier_detection/outlier_detection_step.py", line 166, in process
    step.do_detection()
  File "/Users/jdavies/dev/jwst/jwst/outlier_detection/outlier_detection_spec.py", line 114, in do_detection
    blot_models = self.blot_median(median_model)
  File "/Users/jdavies/dev/jwst/jwst/outlier_detection/outlier_detection.py", line 313, in blot_median
    sinscl=sinscl)
  File "/Users/jdavies/dev/jwst/jwst/resample/gwcs_blot.py", line 78, in extract_image
    interp=interp, exptime=1.0, misval=0.0, sinscl=sinscl)
Exception: NaN in pixmap[0,0]

----------------------------------------------------------------------
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-7-ff84a8a25b8d> in <module>()
      5     '--steps.outlier_detection.skip=False'
      6 ]
----> 7 Step.from_cmdline(args)

~/dev/jwst/jwst/stpipe/step.py in from_cmdline(args)
    160         """
    161         from . import cmdline
--> 162         return cmdline.step_from_cmdline(args)
    163 
    164     @classmethod

~/dev/jwst/jwst/stpipe/cmdline.py in step_from_cmdline(args, cls)
    295             cProfile.runctx("step.run(*positional)", globals(), locals(), profile_path)
    296         else:
--> 297             step.run(*positional)
    298     except Exception as e:
    299         import traceback

~/dev/jwst/jwst/stpipe/step.py in run(self, *args)
    398             else:
    399                 try:
--> 400                     step_result = self.process(*args)
    401                 except TypeError as e:
    402                     if "process() takes exactly" in str(e):

~/dev/jwst/jwst/pipeline/calwebb_spec3.py in process(self, input)
    118 
    119             # Call outlier detection
--> 120             result = self.outlier_detection(result)
    121 
    122             # Resample time. Dependent on whether the data is IFU or

~/dev/jwst/jwst/stpipe/step.py in run(self, *args)
    398             else:
    399                 try:
--> 400                     step_result = self.process(*args)
    401                 except TypeError as e:
    402                     if "process() takes exactly" in str(e):

~/dev/jwst/jwst/outlier_detection/outlier_detection_step.py in process(self, input)
    164             # Set up outlier detection, then do detection
    165             step = detection_step(self.input_models, reffiles=reffiles, **pars)
--> 166             step.do_detection()
    167 
    168             if self.input_container:

~/dev/jwst/jwst/outlier_detection/outlier_detection_spec.py in do_detection(self)
    112             # Blot the median image back to recreate each input image specified
    113             # in the original input list/ASN/ModelContainer
--> 114             blot_models = self.blot_median(median_model)
    115             if save_intermediate_results:
    116                 log.info("Writing out BLOT images...")

~/dev/jwst/jwst/outlier_detection/outlier_detection.py in blot_median(self, median_model)
    311             # apply blot to re-create model.data from median image
    312             blotted_median.data = blot.extract_image(model, interp=interp,
--> 313                                                      sinscl=sinscl)
    314             blot_models.append(blotted_median)
    315 

~/dev/jwst/jwst/resample/gwcs_blot.py in extract_image(self, blot_img, interp, sinscl)
     76         log.info('Blotting {} <-- {}'.format(outsci.shape, self.source.shape))
     77         tblot(self.source, pixmap, outsci, scale=pix_ratio, kscale=1.0,
---> 78                        interp=interp, exptime=1.0, misval=0.0, sinscl=sinscl)
     79 
     80         return outsci

Exception: NaN in pixmap[0,0]
jdavies-st commented 6 years ago

Added a copy of the data and an updated jupyter notebook here

/user/jdavies/issue2395
hbushouse commented 5 years ago

Copied _rate.fits products from original SDP processing, reran calwebb_spec2 to regenerate _cal.fits products, and then ran calwebb_spec3 on association of new cal products, using the latest jwst code (0.14.1dev). Everything runs fine now.

Closing this ticket.